Methods and apparatus for classification

ABSTRACT

A human expert may initially label a white light image of teeth, and computer vision may initially label a filtered fluorescent image of the same teeth. Each label may indicate presence or absence of dental plaque at a pixel. The images may be registered. For each pixel of the registered images, a union label may be calculated, which is the union of the expert label and computer vision label. The union labels may be applied to the white light image. This process may be repeated to create a training set of union-labeled white light images. A classifier may be trained on this training set. Once trained, the classifier may classify a previously unseen white light image, by predicting union labels for that image. Alternatively, the items that are initially labeled may comprise images captured by two different imaging modalities, or may comprise different types of sensor measurements.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/105,382 filed on Aug. 20, 2018, which claims the benefit of U.S.Provisional Application No. 62/547,672 filed Aug. 18, 2017 (the“Provisional”).

FIELD OF TECHNOLOGY

The present invention relates generally to classification.

COMPUTER PROGRAM LISTING

The following six computer program files are incorporated by referenceherein: (1) createpatches_random_train.txt with a size of about 8 KB,created as an ASCII .txt file on Aug. 4, 2018; (2)create_patches_test.txt with a size of about 4 KB, created as an ASCII.txt file on Aug. 4, 2018; (3) interpretResults.txt with a size of about13 KB, created as an ASCII .txt file on Jan. 19, 2018; (4) makeROC.txtwith a size of about 2 KB, created as an ASCII .txt file on Aug. 4,2018; (5) thresholdROCs.txt with a size of about 2 KB, created as anASCII .txt file on Aug. 4, 2018; and (6) vgg_whitelight_cnn.txt with asize of about 13 KB, created as an ASCII .txt file on Aug. 4, 2018.

SUMMARY

In illustrative implementations, union labels (as defined herein) areemployed in training a classifier, such as a convolutional neuralnetwork (CNN). The union labels are calculated by performing a unionoperation.

In some implementations, training a classifier with union labels mayenable the classifier (once it is trained) to exploit knowledge from afirst type of image in order to classify a second type of image. Inother implementations, training a classifier with union labels mayenable the classifier (once it is trained) to exploit knowledge from afirst type of sensor measurement in order to classify a second type ofsensor measurement.

In some implementations, each union label is created by performing aunion operation as follows: A first image of a structure (e.g., tissue)may be captured by a first imaging modality and may be labeled, on aregion-by-region basis, by a human expert. A second image of the samestructure may be captured by a second imaging modality and may belabeled, on a region-by-region basis, by computer vision. In some cases,each region being labeled is a pixel, and thus the labeling for thefirst and second images is done on a pixel-by-pixel basis. In othercases, each region being labeled is a patch of multiple pixels, and thusthe labeling for the first and second images is done on a patch-by-patchbasis. For the first and second images, the classification may bebinary, i.e., may have only two permitted classes. For instance, the twopermitted classes may be 1 and 0, where 1 signifies “condition X ispresent” and 0 signifies “condition X is not present”. The first andsecond images may then be registered, to cause the images to be alignedand to have the same scale. Union labels may then be created in such away that, for each registered region (e.g., pixel or patch) of the twoimages, the union label for that region: (a) is “1” if the label forthat region is “1” in the first image, or in the second image, or inboth the first and second images; and (b) is “0” if the label for thatregion is “0” in both the first and second images.

In some implementations, the union labels are applied, on aregion-by-region basis, to the first set of images which (as notedabove) were captured by the first imaging modality and were initiallylabeled by human experts. A classifier (e.g., a CNN) may be trained onthis union-labeled first set of images. After being trained, theclassifier may classify a previously unseen image captured by the firstimaging modality. When doing so, the trained classifier may, in effect,take into account features in the previously unseen image (captured bythe first imaging modality) which correspond to features in the secondset of images (captured by the second imaging modality). These featuresin the previously unseen image may be so subtle that they would not bedetectable by human experts.

Alternatively, in some implementations, the union labels are applied, ona region-by-region basis, to the second set of images which (as notedabove) were captured by the second imaging modality and were initiallylabeled by computer vision. A classifier (e.g., a CNN) may be trained onthis union-labeled second set of images. After being trained, theclassifier may then classify a previously unseen image captured by thesecond imaging modality. When doing so, the trained classifier may, ineffect, take into account features in the previously unseen image(captured by the second imaging modality) which correspond to featuresin the first set of images (captured by the first imaging modality).

Furthermore, the trained classifier may, when classifying a previouslyunseen image, exploit information contained in the expert labels thatwere used to create the union labels that were in turn employed intraining.

Here is a non-limiting example, in which union labels designate thepresence (or absence) of dental plaque. In this example, two differentsets of images are captured by two different imaging modalities, andthen one set of images is labeled by human experts and the other set ofimages is labeled by computer vision.

Specifically, in this example: (a) the first set of images are whitelight images of teeth; (b) these white light images are captured by anRGB (red green blue) digital camera and are formed by white light; and(c) a human expert labels each pixel in these white light images with alabel that indicates whether dental plaque is (or is not) present in thepixel. For instance, the human expert may use a digital “brush” to“paint” labels on pixels. The human expert may use the “brush” to labela pixel “1” if the expert thinks that dental plaque is present at thepixel and to label the pixel “0” if the expert thinks that dental plaqueis not present at the pixel.

In this example: (a) the second set of images are filtered fluorescentimages (“FF images”) of teeth; (b) these FF images are formed by atleast red fluorescent light that is emitted by porphyrins in response toexcitation illumination and that is filtered by a 530 nm cut-on filter(to remove violet and blue light); (d) the porphyrins are produced bybacteria in dental plaque and thus the red fluorescent light that theporphyrins emit is a visual “signature” of dental plaque; and (d) acomputer vision algorithm labels each pixel in these FF images with alabel that indicates whether dental plaque is (or is not) present in thepixel. For instance, the computer vision algorithm may perform histogramthresholding of each FF image in such a way that: (a) each pixel (ofthat image) which has a recorded light intensity (in a specific colorrange) that exceeds a specified threshold is labeled “1’ signifying“plaque present”; and (b) and each pixel (of that image) which has arecorded light intensity (in that specific color range) that is lessthan or equal to a specified threshold is labeled “0” signifying “plaquenot present”.

In this example, the first and second sets of images are thenregistered. For instance, each pair of images of a particular region ofteeth (consisting of a white light image and a FF image) may beregistered by calculating an affine transformation that includes one ormore of rotation, scaling, and shearing.

In this example, union labels may be created in such a way that, foreach pixel (in a pair of registered images that capture an image of thesame object), the union label for that pixel: (a) is “1” if the labelfor that region is “1” in the white light image, in the FF image, or inboth the white light and FF images; and (b) is “0” if the label for thatregion is “0” in both the white light and FF images.

In this example, the union labels may be applied, on a pixel-by-pixelbasis, to the white light images that were initially labeled by humanexperts. A classifier (e.g., a CNN) may be trained on theseunion-labeled white light images. After being trained on theunion-labeled images, the classifier may then classify a previouslyunseen white light image. When doing so, the trained classifier may, ineffect, take into account features in the previously unseen, white lightimage which correspond to features in the FF (“FF”) images. Humanexperts may ignore these features (e.g., if the human experts would missthe significance of features or if the features are too subtle to beseen by a human eye). Put differently, in this example, the trainedclassifier utilizes (among other things) knowledge acquired from FFimages, when analyzing a previously unseen white-light image.Furthermore, the trained classifier may, when classifying the previouslyunseen white light image, exploit information contained in the expertlabels that were used to create the union labels that were in turnemployed in training.

Alternatively, in this example, the union labels may be applied, on apixel-by-pixel basis, to the FF (“FF”) images that were initiallylabeled by computer vision. A classifier (e.g., a CNN) may be trained onthese union-labeled FF images. After being trained, the classifier maythen classify a previously unseen FF image. When doing so, the trainedclassifier may, in effect, take into account features in the previouslyunseen FF image, which correspond to features in the white light images.Put differently, in this example, the trained classifier utilizes (amongother things) knowledge acquired from white light images, when analyzinga previously unseen FF image. Furthermore, the trained classifier may,when classifying the previously unseen FF image, exploit informationcontained in the expert labels that were used to create the union labelsthat were in turn employed in training.

Alternatively, in this example, each of the labels may indicate thepresence, or absence, of gingivitis.

In the preceding example, the two imaging modalities are white lightimages and filtered red light fluorescent images. However, thisinvention is not limited to these two specific imaging modalities.

In some implementations, any two imaging modalities may capture pairs ofimages, where each pair consists of two images of a particular object(e.g. a region of tissue), one image being captured by the first imagingmodality and the second image being captured by the second imagingmodality. These pairs of images may then be employed to create unionlabels, and then union-labeled images may be employed to train aclassifier.

In some cases, the two imaging modalities comprise (a) an imagingtechnology with a specific contrast agent and (b) the same imagingtechnology without a contrast agent. Or the two imaging modalities maycomprise (a) an imaging technology with a first contrast agent and (b)the same imaging technology with a second contrast agent. Or the twoimaging modalities may comprise (a) an imaging technology with aspecific tissue stain or dye; and (b) the same imaging technologywithout a tissue stain or dye. Or the two imaging modalities maycomprise (a) an imaging technology with a first tissue stain or dye; and(b) the same imaging technology with a second tissue stain or dye. Orthe two imaging modalities may comprise (a) an imaging technology with afirst radionuclide and (b) the same imaging technology with a secondradionuclide. Or the two imaging modalities may comprise (a) an imagingtechnology with a first radiotracer and (b) the same imaging technologywith a second radiotracer. Or the two imaging modalities may comprise(a) an imaging technology with a first radioligand and (b) the sameimaging technology with a second radioligand. Or the two imagingmodalities may comprise two different imaging technologies which employdifferent hardware (e.g., Mill and CT).

In some implementations, the two images in a pair of correspondingimages (one captured by the first imaging modality and the othercaptured by the second imaging modality) are each images of the sameregion of tissue.

In some cases, the two images which are registered are both 2D(two-dimensional) images. For instance, a 2D white light image of anunstained tissue biopsy slide may be registered with a 2D white lightimage of the same biopsy slide after the tissue has been stained withH&E (hematoxylin and eosin) stain. Or, for instance, a 2D PET (positronemission tomography) image of a “slice” of tissue may be registered witha 2D Mill (magnetic resonance imaging) image of the same “slice” of thesame tissue. This registration may be performed in the same way thatregistration is performed in existing PET-MRI systems. Or, for instance,a 2D PET image of a “slice” of tissue may be registered with a 2D CT(x-ray computed tomography) image of the same “slice” of the sametissue. This registration may be performed in the same way thatregistration is performed in existing PET-CT systems.

Alternatively, in some cases, the two images which are registered areboth 3D (three-dimensional) images. For instance, a 3D PET image of a 3Dregion of tissue may be registered with a 3D MRI image of the same 3Dregion of the same tissue. Or, for instance, a 3D PET image of a 3Dregion of tissue may be registered with a 3D CT image of the same 3Dregion of the same tissue.

In some cases, a classifier (which has been trained with union-labeledimages): (a) determines that a region of an image is a member of aclass; or (b) calculates a probability that a region of an image is amember of a class. Alternatively, a classifier (which has been trainedon union labels) may calculate a degree of membership (in fuzzy setterminology) regarding the degree to which a region of an image is amember of a fuzzy set.

In some cases, a classifier (which has been trained with union-labeledimages) performs a binary classification. For instance, in some cases:(a) the classifier classifies a region of an image to be either “X” or“not X”, where these are the only two permitted classes; or (b) theclassifier determines the probability that the region is “X”, where theonly permitted classes are “X” and “not X”.

In some cases, a classifier (which has been trained with union-labeledimages) classifies with more than two permitted classes. For instance,in some cases: (a) the classifier classifies a region of an image to beeither “X”, “Y” or “Z”, but only one of these classes at a time; or (b)the classifier determines the probability that the region is “X”, theprobability that the region is “Y” or the probability that the region is“Z”, where the only permitted classes are “X”, “Y” and “Z”.

In some cases, a classifier (which has been trained with union-labeledimages) classifies an item as being simultaneously in multiple classes.For instance, in some cases, the classifier classifies a region of animage to be simultaneously both “X” and “Y”.

In some cases, more than two imaging modalities are employed. Forinstance, there may be two, three, four, five, six or more imagingmodalities.

In some implementations of this invention: (a) X different types ofimaging modalities are employed, where X≥2; (b) registration isperformed in such a way that, after registration, the images captured bythe X types of imaging modalities comprise groups of registered imageswhere all of the images in each particular group are registered witheach other and are images of the same physical object; (c) each group ofregistered images includes X images, one per type of imaging modality;and (d) for each specific group of registered images, the union labelfor a region (e.g., a pixel or patch) is the union of the initial labelsfor that region. For instance, in some cases, if a group of registeredimages includes images captured by only three types of imagingmodalities and the initial labels for a specific pixel (in that group ofregistered images) are “1” for the first imaging modality, “0” for thesecond imaging modality, and “0” for the third imaging modality, thenthe union label for that specific pixel is “1”. Likewise, in some cases,if a group of registered images includes images captured by only threetypes of imaging modalities and the initial labels for a particularpixel (in that group of registered images) are “0” for the first imagingmodality, “0” for the second imaging modality, and “0” for the thirdimaging modality, then the union label for that particular pixel is “0”.In some cases: (a) images captured by at least one of the X types ofimaging modalities are initially labeled by a human expert; and (b)images captured by at least one other of the X types of imagingmodalities are initially labeled by a computer algorithm (e.g., acomputer vision algorithm).

In some implementations, each union label is calculated as the inclusivedisjunction (or equivalently, Boolean OR) of multiple other labels. Forinstance, in some cases: (a) the only permitted values of a first labelare “0” and “1” and the only permitted values of a second label are “0”and “1”; and (b) a union label is calculated as being equal to theinclusive disjunction (or, equivalently, Boolean OR) of the first andsecond labels.

This invention is a major improvement to prior technology. Inillustrative implementations, this invention solves at least thefollowing two technological problems: (1) the inability of conventionaltechnology to gain the benefit of information acquired from imagescaptured by a first imaging modality, while viewing only a previouslyunseen image captured by a second imaging modality, where the firstimaging modality is different than the second imaging modality; and (2)the inability of conventional technology to gain the benefit ofinformation acquired from measurements by a first type of sensor, whenpresented with only a previously unseen measurement by a second type ofsensor, where the first type of sensor is different than the second typeof sensor. In some implementations, this invention solves these twotechnological problem by employing a union-label trained classifier, asdescribed herein.

As used herein, a “pre-training image” means an image, the labels ofwhich are used to help create union labels.

This invention has many practical benefits such as, in some cases (a)using only a low radiation dose imaging method, while exploitinginformation acquired from pre-training images captured by a highradiation dose imaging method; (b) using only a less expensive imagingmethod, while exploiting information acquired from pre-training imagescaptured by a more expensive imaging method; or (c) using only a morewidely available imaging method, while exploiting information acquiredfrom a more scarcely available imaging method.

Reducing Exposure to Ionizing Radiation: In some implementations, imagescaptured by a first imaging modality (which exposes a patient to littleor no ionizing radiation) are classified in a way that exploitsknowledge of pre-training images captured by another imaging method(which exposes patients to a higher dose of ionizing radiation). Thus,in some implementations of this invention, a patient gains the benefitof pre-training images captured by the imaging method with the higherionizing radiation, without being exposed to that higher radiationimaging method. For instance, in some cases: (a) a CT scan would exposea patient to a higher dose of x-ray radiation than does an MRI; (b) aclassifier is trained with union-labeled MM images, where each unionlabel is the union of a label for a region of a CT image and a label forthe same region in an MRI image; (c) a new MRI image is captured; and(d) the trained classifier is able to classify this new MRI image, in amanner that exploits knowledge about pre-training images captured by CT.Thus, due to the trained classifier (which has been trained withunion-labeled MRI images), the patient may get much of the benefit of aCT scan, without exposing the patient to the higher x-ray dose of a CTscan.

Likewise, in some implementations, a white light image of the surface ofa tooth may be registered with an x-ray image of the same tooth. Forinstance, in some cases: (a) an x-ray of a tooth would cost more than awhite light image of the tooth and would expose the patient to moreionizing radiation; (b) a classifier is trained with union-labeled whitelight images of teeth, where each union label is the union of a labelfor a region of a white light image and a label for the same region inan x-ray image; (c) a new white light image of a tooth is captured; and(d) the trained classifier is able to classify this white light image ofa tooth, in a manner that exploits knowledge about pre-training x-rayimages. Thus, due to the trained classifier (which has been trained withunion-labeled white light images), the patient may get much of thebenefit of a dental x-ray, at a lower cost and without exposing thepatient to the ionizing radiation of a dental x-ray.

Reducing Cost and Need for Human Experts: In some implementations, anew, unlabeled image captured by a first imaging modality is classifiedin a way that exploits knowledge: (a) of pre-training images captured byanother imaging method; and (b) of expert labels applied to pre-trainingimages captured by the first imaging modality. For example, in somecases: (a) white light images of teeth are labeled by human experts; (b)this human expert labeling is expensive and difficult to provide on amass scale; (c) FF images of teeth are initially labeled by computervision; (d) a classifier is trained with union-labeled white lightimages, where each union label is the union of a human expert label fora pixel of a white light image and computer vision label for the samepixel in a FF image; (e) a new white light image is captured; and (f)the trained classifier classifies this new white light image. In theexample in the preceding sentence, the patient may get much of thebenefit of a FF image and of the expert labels, even though the trainedclassifier is classifying a new white light image that has not beenlabeled by a human expert.

In some implementations, union labels may be used to great advantagewhere (i) multiple imaging modalities are employed in order to createunion labels, (b) each imaging modality is an independent way to detecta condition and (c) there is little overlap between the “positives”(presence of the condition) detected by the different imagingmodalities. For instance, in the dental plaque example above, in somecases: (a) the FF images and white light images each detect someinstances of plaque, but only a small minority of instances of plaqueare detected by both the FF images and white light images; (b) the FFimages do not detect a first subset of the plaque, because the firstsubset contains little or no porphyrins that emit red fluorescent light;and (c) the expert-labeled white light images do not detect a secondsubset of the plaque. Thus, in this dental plaque example, it isadvantageous to train with union-labeled images, where the union labelfor a pixel is “positive” for plaque if either the white light image, FFimage or both detect plaque at that pixel. Likewise, in someimplementations, union labels may be used to great advantage where (i)multiple types of sensors are employed in order to create union labels,(b) each type of sensor is an independent way to detect a condition and(c) there is little overlap between the “positives” (presence of thecondition) detected by the different types of sensors.

In some implementations, this invention may be employed: (a) to diagnosedisease; (b) to perform patient stratification; or (c) to analyze orpredict drug impact.

The Summary and Abstract sections and the title of this document: (a) donot limit this invention; (b) are intended only to give a generalintroduction to some illustrative implementations of this invention; (c)do not describe all of the details of this invention; and (d) merelydescribe non-limiting examples of this invention. This invention may beimplemented in many other ways. Likewise, the Field of Technologysection is not limiting; instead it identifies, in a general,non-exclusive manner, a field of technology to which someimplementations of this invention generally relate.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows union labels that are the union of human expert labels andcomputer vision labels.

FIG. 2 shows a flowchart for a method that employs union-labeled whitelight images to train a classifier, in such a way that the trainedclassifier may accurately classify a previously unseen white lightimage.

FIG. 3 shows a flowchart for a method that employs union-labeledbiomarker images to train a classifier, in such a way that the trainedclassifier may accurately classify a previously unseen biomarker image.

FIG. 4 shows a flowchart for a method that employs union-labeled“modality A” images to train a classifier, in such a way that thetrained classifier may accurately classify a previously unseen “modalityA” image.

FIG. 5 shows a flowchart for a method that employs a classifier (whichhas been trained with union-labeled images) to perform binaryclassification.

FIG. 6 shows a flowchart for a method that employs a classifier (whichhas been trained with union-labeled images) to classify in more than twoclasses.

FIG. 7 shows a flowchart for a method that employs a classifier (whichhas been trained with union-labeled images) to classify an object asbeing simultaneously in multiple classes.

FIG. 8 shows hardware of an imaging system that is configured to acquireand classify an image.

FIG. 9 shows a flowchart for a method that employs union-labeled sensordata to train a classifier, in such a way that the trained classifiermay accurately classify previously unseen sensor data.

FIG. 10 shows hardware of a sensor system that is configured to acquireand classify sensor data.

FIG. 11 shows a flowchart for a method that employs union-labeled datato train a classifier, in such a way that the trained classifier mayaccurately classify previously unseen data.

The above Figures are not necessarily drawn to scale. The above Figuresshow some illustrative implementations of this invention, or provideinformation that relates to those implementations. The examples shown inthe above Figures do not limit this invention. This invention may beimplemented in many other ways.

DETAILED DESCRIPTION

Union Labels

FIG. 1 shows union labels that are the union of human expert labels andcomputer vision labels.

In the example shown in FIG. 1, dental plaque is present on portions ofthe surface of teeth. Human experts are often able to detect this dentalplaque in white light images of the teeth, such as the white lightimages shown in columns 101 and 106 of FIG. 1. When stimulated by a 405nm excitation light, porphyrins produced by this dental plaque emit redfluorescent light that may be filtered (to remove blue and violet light)and captured in filtered fluorescent (“FF”) images, such as the FFimages in columns 103 and 108 of FIG. 1. Thus, the red fluorescent lightemitted by the porphyrins is a signature of (visual indicator for)dental plaque.

In FIG. 1, column 101 shows white light images of teeth. Column 102shows labels that were applied by human experts to the white lightimages in column 101. Column 103 shows FF images of the same teeth as incolumn 101. Column 104 shows labels that were applied by computer visionto the FF images in column 103. Column 105 shows union labels that arethe union of human expert labels in column 102 and computer visionlabels in column 104.

Likewise, in FIG. 1, column 106 shows white light images of teeth.Column 107 shows labels that were applied by human experts to the whitelight images in column 106. Column 108 shows FF images of the same teethas in column 106. Column 109 shows labels that were applied by computervision to the FF images in column 108. Column 110 shows union labelsthat are the union of human expert labels in column 107 and computervision labels in column 109.

In FIG. 1, columns 102, 104, 105, 107, 109, 110 each show binary labels,where white (or “1”) means “dental plaque present” and black (or “0”)means “dental plaque not present”.

In FIG. 1, union labels are applied to each pair of registered images.The union labels are calculated in such a way that: (a) a union labelapplied to a pixel is black (indicating plaque not present) if both theexpert label and computer vision label for that pixel are black; and (b)a union label applied to a pixel is white (indicating plaque present) ifthe expert label for that pixel is white, or if the computer visionlabel for that pixel is white, or if both the expert and computer visionlabels for that pixel are white.

In FIG. 1, the union pixels are applied on a pixel-by-pixel basis, oneunion label for each image pixel. Alternatively, the union labels may beapplied on a patch-by-patch basis, where each patch is a connected groupof two or more pixels.

In any particular row of images in FIG. 1: (a) the teeth shown in column101 in that row are the same as the teeth shown in column 103 in thatrow (before registration); (b) the teeth shown in column 106 in that roware the same as the teeth shown in column 108 in that row (beforeregistration); (c) the union labels shown in column 105 in that row arethe union of the human expert labels shown in column 102 in that row andthe computer vision labels shown in column 104 in that row; and (c) theunion labels shown in column 110 in that row are the union of the humanexpert labels shown in column 107 in that row and the computer visionlabels shown in column 109 in that row.

In the example shown in FIG. 1, the plaque regions that the humanexperts detect overlap only partially with the plaque regions that thecomputer vision detects. Thus, in FIG. 1, a large proportion of theplaque regions that the human experts detect is not detected by computervision, and vice versa. In FIG. 1, each union label labels a toothregion as containing plaque if a human expert, or computer vision, orboth, has (or have) identified the region as containing plaque.

Alternatively, in FIG. 1: (a) gingivitis may be detected, instead ofdental plaque; and (b) each label may indicate the presence of, orabsence of, gingivitis.

Employing Union Labels

FIG. 2 shows a flowchart for a method that employs union-labeled whitelight images to train a classifier, in such a way that the trainedclassifier may accurately classify a previously unseen white lightimage.

FIG. 3 shows a flowchart for a method that employs union-labeledbiomarker images to train a classifier, in such a way that the trainedclassifier may accurately classify a previously unseen biomarker image.

In FIGS. 2 and 3, a white light camera 201 captures a white light image202 of object 200. The white light image 202 is formed by white light inthe visible spectrum. A human expert creates labels 203 for the whitelight image, on a pixel-by-pixel basis. Each expert label indicateseither that “X is present” or “X is not present” at the pixel to whichthe label is applied. In FIGS. 2 and 3, a biomarker camera 204 capturesa biomarker image 205 of object 200. The biomarker image 205 is capturedby a different imaging modality than that employed to capture whitelight image 202. Thresholding 206 is performed to detect pixels thathave a parameter (e.g., incident light intensity) that exceeds (oralternatively, is less than) a specified threshold. A computer visionalgorithm creates labels 207 for the biomarker image, on apixel-by-pixel. Each computer vision label indicates either that “X ispresent” or “X is not present” at the pixel to which the label isapplied.

In FIGS. 2 and 3, registration 208 is performed to register the whitelight image 202 and biomarker image 205. A computer calculates the union209 of the expert labels and the computer vision-generated labels on apixel-by-pixel basis, so that there is a union label for each pixel.

In the example shown in FIG. 2, the union labels are applied to whitelight image 202. The resulting union-labeled white light image 210 is atraining image.

In FIG. 2, a set of training images is created by repeating the aboveprocess (described in the preceding three paragraphs) for multipleobjects. For instance, if object 200 comprises a tooth, then manydifferent teeth may be imaged, and for each different tooth, the processdescribed (described in the preceding three paragraphs) may be performedto create a training image.

In FIG. 2, training 211 is performed to train a classifier on a set oftraining images, where each training image is union-labeled white lightimage. The trained classifier 212 then classifies a previously unseenwhite light image 220 by predicting union labels 221 for the previouslyunseen white light image.

In the example shown in FIG. 3, the union labels are applied tobiomarker image 205. The resulting union-labeled biomarker image 310 isa training image.

In FIG. 3, a set of training images is created by repeating the aboveprocess (described above with respect to 201-209 and 310 in FIGS. 2 and3) for multiple objects. For instance, if object 200 comprises a tooth,then many different teeth may be imaged, and for each different tooth,this process (described above with respect to 201-209 and 310 in FIGS. 2and 3) may be performed to create a training image.

In FIG. 3, training 311 is performed to train a classifier on a set oftraining images, where each training image is union-labeled biomarkerimage. The trained classifier 312 then classifies a previously unseenbiomarker image 320 by predicting union labels 321 for the previouslyunseen biomarker image.

In some cases, in FIGS. 2 and 3: (a) object 200 comprises teeth; (a)biomarker image 205 is formed by at least red fluorescent light that isemitted by porphyrins on the surface of the teeth and that is filteredto remove blue and violet light; (b) biomarker camera 204 is a camerathat is configured to emit 405 nm light (which excites thefluorescence), to filter out blue and violet light and to capture redlight and other light in the visible spectrum; (c) thus, biomarkercamera 204 captures red fluorescent light which is emitted by porphyrinsand is a signature for dental plaque on the teeth; and (d) a computervision algorithm detects pixels in the biomarker image that correspondto the presence of dental plaque, by histogram thresholding to determinea set of pixels that have a measured intensity of incident light (in aspecific color range) above a specified threshold, and labeling thosepixels as “plaque present” and labeling all other pixels “plaque notpresent”.

In some cases, in FIG. 2, the trained classifier 212 classifies apreviously unseen white light image 220 of teeth, by predicting, on apixel-by-pixel basis, union labels 221 for that previously unseen whitelight image 220.

In some cases, in FIG. 3, the trained classifier 312 classifies apreviously unseen biomarker image 320 of teeth, by predicting, on apixel-by-pixel basis, union labels 321 for that previously unseenbiomarker image 220

In some cases, in FIGS. 2 and 3, each label (out of labels 221 and 321)specifies whether, at the pixel to which the label applies, dentalplaque is or is not present. In some cases, in FIGS. 2 and 3, each label(out of labels 221 and 321) that is applied to a pixel specifies aprobability that plaque is present at that pixel. In some cases, inFIGS. 2 and 3, each label (out of labels 221 and 321) that is applied toa pixel specifies a degree to which plaque is present at that pixel (orspecifies a relative amount of plaque that is present at that pixel).

Alternatively, in some cases, in FIGS. 2 and 3, each label (out oflabels 221 and 321) specifies whether, at the pixel to which the labelapplies, gingivitis is or is not present. In some cases, in FIGS. 2 and3, each label (out of labels 221 and 321) that is applied to a pixelspecifies a probability that gingivitis is present at that pixel. Insome cases, in FIGS. 2 and 3, each label (out of labels 221 and 321)that is applied to a pixel specifies a degree to which gingivitis ispresent at that pixel (or specifies a relative amount of gingivitis thatis present at that pixel).

In some cases, in FIGS. 2 and 3, labels are applied to biomarker images202 by human experts, instead of by computer vision. For instance, thehuman expert may label biomarker images by putting boxes aroundbiomarkers in the biomarker images. For example, in some cases: (a)biomarker images 202 are fluorescent images; (b) fluorescent regions arevisible in these images; and (c) a human expert puts boxes around thefluorescent regions. In these cases, the phrase “computer-visiongenerated biomarker labels” for 207 may be replaced by the phrase“expert biomarker labels.”

FIG. 4 shows a flowchart for a method that employs union-labeled“modality A” images to train a classifier, in such a way that thetrained classifier may accurately classify previously unseen “modalityA” images.

In some cases, in FIG. 4, the “modality A” imaging system differs, in astructural aspect or in a functional aspect, from the “modality B”imaging system. Thus, in some cases, in FIG. 4, the modality B image 405is captured by a different imaging modality than that employed tocapture modality A image 402.

In the example shown in FIG. 4, a modality A camera 401 captures amodality A image 402 of object 400. Labels 403 may be applied to themodality A image, on a pixel-by-pixel basis. In FIG. 4, a modality Bcamera 404 captures a modality B image 405 of object 400. Labels 407 maybe applied to the modality B image, on a pixel-by-pixel. Each label (outof labels 403 and 407) may classify a pixel as either “X” or “not X”.

In some cases, labels 403 are created by a human expert and labels 407are generated by a computer vision algorithm. In some cases, labels 403are generated by a computer vision algorithm and labels 407 are createdby a human expert. In some cases, labels 403 and 407 are all created byone or more human experts. In some cases, labels 403 and 407 are allgenerated by a computer vision algorithm.

In FIG. 4, registration 408 is performed to register the modality Aimage 402 and modality B image 405. A computer may calculate the union409 of the expert labels and the union labels on a pixel-by-pixel basis,so that there is a union label for each pixel. The union labels may beapplied to modality A image 402. The resulting union-labeled modality Aimage 410 may comprise a training image.

A set of training images may be created by repeating the process(described in the preceding three paragraphs) for multiple objects. Forinstance, if object 400 comprises a tissue structure, then manydifferent examples of that tissue structure may be imaged, and for each,the process (described in the preceding three paragraphs) may beperformed to create a training image.

Training 411 may be performed to train a classifier on a set of trainingimages, where each training image is union-labeled modality A image.Then, the trained classifier 412 may classify a previously unseenmodality A image 420 by predicting union labels 421 for the previouslyunseen modality A image.

In FIG. 4, imaging modality A may comprise an imaging technology with aspecific contrast agent and imaging modality B may comprise the sameimaging technology without a contrast agent. Or imaging modality A maycomprise an imaging technology with a first contrast agent and imagingmodality B may comprise the same imaging technology with a secondcontrast agent. Or imaging modality A may comprise an imaging technologywith a specific tissue stain or dye; and imaging modality B may comprisethe same imaging technology without a tissue stain or dye. Or imagingmodality A may comprise an imaging technology with a first tissue stainor dye; and imaging modality B may comprise the same imaging technologywith a second tissue stain or dye. Or imaging modality A may comprise animaging technology with a first radionuclide and imaging modality B maycomprise the same imaging technology with a second radionuclide. Orimaging modality A may comprise an imaging technology with a firstradiotracer and imaging modality B may comprise the same imagingtechnology with a second radiotracer. Or imaging modality A may comprisean imaging technology with a first radioligand and imaging modality Bmay comprise the same imaging technology with a second radioligand.

In FIG. 4, in some cases, imaging modalities A and B may comprise twodifferent imaging technologies which employ different hardware (e.g.,MRI and CT). For instance, in some cases, imaging modality A and imagingmodality B may capture: (a) white light images and x-ray images,respectively; (b) white light images and PET images, respectively; (c)white light images and MRI images, respectively; (d) MM images and CTimages, respectively; or (e) white light images and CT images,respectively.

For instance, each of the following is a different imaging modality, forpurposes hereof: (1) MRI (magnetic resonance imaging) without a contrastagent; (2) MRI with gadoterate contrast agent; (3) MRI with gadodiamidecontrast agent; (4) MRI with gadobenate contrast agent; (5) MRI withgadopentetate contrast agent; (6) MRI with gadoteridol contrast agent;(7) MRI with gadoversetamide contrast agent; (8) MM with gadoxetatecontrast agent; (9) MM with gadobutrol contrast agent; (10) MM withgadofosveset contrast agent; (11) MRI with gadocoletic acid contrastagent; (12) MM with gadomelitol contrast agent; (13) MM withsuperparamagnetic iron oxide contrast agent; (14) MRI with manganesechelate contrast agent; (15) MRI with β-galatosidase-activated contrastagent; (16) PET (positron emission tomography) with a radiotracercomprising FDG (fluorodeoxyglucose) labeled with flourine-18; (17) PETwith oxygen-15 water radiotracer; (18) PET with nitrogen-13 ammoniaradiotracer; (19) PET with rubidium-82 chloride radiotracer; (20) PETwith carbon-11 acetate radiotracer; (21) SPECT (single-photon emissioncomputed tomography) with technetium-99m radioisotope; (22) SPECT withan iodine radioisotope; (23) SPECT with indium-111 radioisotope; (24) CT(x-ray computed tomography) without contrast agent; (25) CT with iohexolcontrast agent; (26) CT with iopromide contrast agent; (27) CT withiothalamate contrast agent; (28) CT with ioxaglate contrast agent; (29)CT with iodixanol contrast agent; (30) CT with iopamidol contrast agent;(31) CT with iosimenthol contrast agent; (32) CT with GE-145 contrastagent; (33) ultrasound without contrast agent; (34) ultrasound withbubble contrast agent comprising an albumin shell with anoctafluoropropane gas core; (35) ultrasound with bubble contrast agentcomprising a lipid/galactose shell with an air core; (36) ultrasoundwith a bubble contrast agent comprising a lipid shell with anoctafluoropropane gas core; (37) ultrasound with any other bubblecontrast agent comprising a different combination of (i) chemicalcomposition of bubble shell and (ii) chemical composition of gas core;(38) white light image of tissue (e.g., tissue biopsy) without anystain; (39) white light image of tissue stained by hematoxylin and eosin(H&E) stain; (40) white light image of tissue stained by Toluidine bluestain; (41) white light image of tissue stained by Masson's trichromestain; (42) white light image of tissue stained by Mallory's trichromestain; (43) white light image of tissue stained by Weigert's elasticstain; (44) white light image of tissue stained by Heidenhain's AZANtrichrome stain; (45) white light image of tissue stained by Silverstain; (46) white light image of tissue stained by Wright's stain; (47)white light image of tissue stained by Orcein stain; (48) white lightimage of tissue stained by Periodic acid-Schiff stain (PAS) stain; (49)x-ray projection radiograph; and (50) fluorescent image. For each pairof these fifty different imaging modalities, a first image (captured byone of the imaging modalities in the pair) and a second image (capturedby the second imaging modality in the pair) comprise a modality A imageand a modality B image, respectively, as those terms are used herein.This paragraph and the preceding two paragraphs are not an exhaustivelist of imaging modalities that may be employed. This paragraph and thepreceding two paragraphs are not an exhaustive list of modality A imagesand modality B images. This paragraph and the preceding two paragraphslist non-limiting examples. Other imaging modalities may be employed inthis invention.

In some cases, in FIGS. 2, 3 and 4, trained classifier 212, 312, 412comprises a neural network, such as a convolutional neural network(CNN). As noted above, in FIGS. 2, 3 and 4, labels are applied on apixel-by-pixel basis. Alternatively, FIGS. 2, 3 and 4, labels may beapplied on a patch-by-patch basis, where each patch is a group ofmultiple pixels.

In some implementations of this invention, a CNN performs patch-basedimage analysis, in which each patch of pixels (e.g., a patch of 21×21pixels) is analyzed in order to classify a pixel at the center of thepatch. This may be done in such a way that a different patch is analyzedfor each pixel being labeled. Alternatively, instead of a patch-basedapproach, the image analysis may employ a pixel-based approach forexample using an FCN, or a region-based approach for example using anR-CNN.

In some implementations, an output of the neural network (that wastrained on images labeled with union labels) is a label that classifiesan entire image. Alternatively, in some implementations, an output ofthe neural network (that was trained on images labeled with unionlabels) comprises: (a) a set of labels, each of which labels a differentregion of an image, or (b) a label that labels a set of images.

In some implementations, a single human expert or a group of humanexperts annotates images, to create the human-annotated images that aretaken as an input when calculating union labels. Alternatively, in somecases, the human annotations are provided by non-experts throughcrowdsourcing.

In illustrative implementations of this invention, a wide variety ofcolor models may be employed in images that are classified. Among otherthings, the RGB (Red Green Blue) color model or the HSI (Hue SaturationIntensity) color model may be employed. In some case, a camera capturesRGB data, and this data is transformed into HSI data. In someimplementations, the same color model is used for all of the classifiers(e.g., RGB may be employed in images classified by initial machineclassifier, images annotated by a human expert, training images that arelabeled with union labels, and in a previously unknown image that isbeing classified by a trained CNN).

In some implementations, a set of images that are labeled compriseimages of teeth, dental plaque, dental fluorosis, other dental or oraldiseases or conditions, images of biopsy tissue samples, or images oforgans, tissue, or diseases, or other medical images.

In some cases, at least one of the sets of images (e.g., a set of imagesinitially classified by computer vision or initially annotated by ahuman expert) comprises images of fluorescent porphyrin biomarkers, orother fluorescent biomarkers such as pyridoxal, siderophore (e.g.,pyoverdin) or other fluorophores.

In some implementations of this invention, at least one of the sets ofimages (e.g., a set of images initially classified by machine orinitially annotated by a human) comprises images of light in the visiblespectrum. In some implementations, a wide variety of technology may beemployed to capture light in the visible spectrum. For example, in someimplementations of this invention, images of visible light may becaptured by a digital single-lens reflex camera, point-and-shoot camera,webcam, camera housed in a smartphone, digital camera, digital videocamera, plenoptic camera, fluorescent biomarker imaging camera, CCD(charge-coupled device) camera, CMOS (complementarymetal-oxide-semiconductor) camera, NMOS (N-typemetal-oxide-semiconductor) camera, Live MOS camera, flat panel detector,or other camera.

This invention is not limited to classifying images of light captured inthe visible spectrum. For example, in some cases, at least one set ofimages (e.g., a set of images initially classified by machine orinitially annotated by a human) may comprise any type of imaging dataacquired at any wavelength of light, such as data measured bynear-infrared imaging, x-ray, CT (x-ray computed tomography), cone-beamCT, infrared imaging, terahertz imaging (such as terahertz time-domainspectroscopy), gigahertz imaging, time-of-flight imaging, radar, LIDAR,Mill (magnetic resonance imaging), or multi-spectral imaging.

In some alternative implementations of this invention: (a) there is onlyone imaging modality and only one imaging device; (b) each image, in aset of images captured by the imaging device, is labeled by a humanexpert and is also labeled by a computer vision algorithm; (c) unionlabels are generated, where the union label for each specific pixel isthe union of the expert label and the computer-vision generated label;(d) steps (b) and (c) of this sentence are repeated to create a set ofunion-labeled training images; (e) a classifier is trained with this setof union-labeled training images; and (f) after being trained, thetrained classifier may classify a previously unseen image captured bythe imaging device, in order to produce union labels for the previouslyunseen image.

Classification

FIG. 5 shows a flowchart for a method that employs a classifier (whichhas been trained with union-labeled images) to classify with a binaryclassification.

FIG. 6 shows a flowchart for a method that employs a classifier (whichhas been trained with union-labeled images) to classify in more than twoclasses.

FIG. 7 shows a flowchart for a method that employs a classifier (whichhas been trained with union-labeled images) to classify an object asbeing simultaneously in multiple classes.

In FIGS. 5, 6, and 7, a multi-pixel region 501 is an input to a neuralnetwork 502. In some cases, multi-pixel region 501 is a patch of pixels,such as a patch of 21×21 pixels. Alternatively, multi-pixel region 501may be an entire image. In some cases, neural network 502 is aconvolutional neural network. For instance, neural network 502 maycomprise the first 13 layers of a VGG16 convolutional network followedby a fully-connected layer and soft max function (a “modified VGG16”).Each of the convolution layers in this modified VGG16 architecture mayhave a ReLU (rectified linear unit) activation function. The layers ofthis modified VGG16 are depicted in FIGS. 5, 6 and 7. Alternatively,neural network 502 may have an architecture that is different than amodified VGG16.

In the example shown in FIG. 5, neural network 502 (which has beentrained with union-labeled images) outputs a binary classification. Forinstance, in FIG. 5, neural network 502: (a) may take, as an input, apatch of pixels in an image (such as a patch of 21×21 pixels); and (b)may output a binary classification of a pixel in the center of thepatch. For example, in FIG. 5, neural network 502: (a) may classify thecentral pixel as “X” or “not X”, but not both at the same time, where“X” and “not X” are the only two classes permitted in theclassification; or (b) may calculate the probability that the centralpixel is “X”, where the only permitted classes are “X” and “not X”. Or,in FIG. 5, neural network 502 may determine the central pixel's degreeof membership in two classes (“X” and “not X”), where “X” and “not X”are the only two classes that the patch is permitted to be a member of(at least to some degree).

In the example shown in FIG. 6, neural network 502 (which has beentrained with union-labeled images) outputs an “N-ary”classification—that is, a classification in which there are N permittedclasses, where N≥3. For instance, in FIG. 6, neural network 502: (a) maytake, as an input, a patch of pixels in an image (such as patch of 21×21pixels); and (b) may output an N-ary classification of a pixel in thecenter of the patch. For example, in FIG. 6, neural network 502: (a) mayclassify the central pixel as “X”, “Y” or “Z”, but only one at a time,where “X”, “Y” and “Z” are the only classes permitted in theclassification; or (b) may calculate the probability that the centralpixel is “X”, the probability that the central pixel is “Y”, and theprobability that the central pixel is “Z”, where the only threepermitted classes are “X”, “Y” and “Z”. Or, in FIG. 6, neural network502 may determine the central pixel's degree of membership in threeclasses “X”, “Y” and “Z”, where “X”, “Y” and “Z” are the only threeclasses that the patch is permitted to be a member of (at least to somedegree).

In FIG. 6, union labels for each permitted class (out of the three ormore permitted classes) may be computed separately. For instance, inFIG. 6, if the only three permitted classes are “X”, “Y” and “Z”, then:(a) the union labels for “X” may be computed separately; (b) the unionlabels for “Y” may be computed separately; and (c) (a) the union labelsfor “Z” may be computed separately.

In the example shown in FIG. 7, neural network 502 (which has beentrained with union-labeled images) classifies an object as beingsimultaneously in multiple classes. For instance, in FIG. 7, neuralnetwork 502: (a) may take, as an input, a patch of pixels in an image(such as a patch of 21×21 pixels); and (b) may classify the pixel in thecenter of the patch as being simultaneously in multiple classes. Forexample, in FIG. 7, if the central pixel may simultaneously be a memberof one or more of “X”, “Y” and “Z”, and if “X”, “Y” and “Z” are the onlypermitted classes (other than combinations of permitted classes), thenneural network 502: (a) may classify the central pixel as being “X”,being “Y”, being “Z”, being “X and Y”, being “Y and Z”, being “X and Z”,or being “X, Y and Z”; or (b) may calculate the probability that thecentral pixel is “X”, the probability that the central pixel is “Y”, theprobability that the central pixel is “Z”, the probability that thecentral pixel is “X and Y”, the probability that the central pixel is “Yand Z”, the probability that the central pixel is “X and Z”, and theprobability that the central pixel is “X, Y and Z”.

In FIG. 7, union labels for each permitted class (or group of classes)may be computed separately. For instance, in FIG. 7, if an object maysimultaneously be a member of one or more of “X”, “Y” and “Z”, then: (a)the union labels for class “X” may be computed separately; (b) the unionlabels for class “Y” may be computed separately; (c) the union labelsfor class “Z” may be computed separately; (d) the union labels for class“X and Y” may be computed separately; (e) the union labels for class “Yand Z” may be computed separately; (f) the union labels for class “X andZ” may be computed separately; and (b) the union labels for class “X, Yand “Z” may be computed separately.

As noted above, in some implementations of this invention, the labelsindicate degree of membership in a fuzzy set. For instance, a pixel'sdegree of membership in a fuzzy set called “dental plaque” may indicatethe relative amount of plaque that is present at that pixel. Forexample, 1.0 may indicate heavy plaque, 0.5 may indicate medium plaqueand 0.0 may indicate no plaque.

In some implementations (where the expert labels and the initialcomputer vision-generated labels denote degree of membership in a fuzzyset), a fuzzy union label may be calculated in different ways. Forpurposes of this paragraph, let a∈[0,1] be the degree of membership of xin a fuzzy set A; let b∈[0,1] be the degree of membership of x in afuzzy set B; and let U(a, b) be the fuzzy union of a and b. Forinstance, x may be a pixel in an image, or a patch of pixels in animage. In some implementations of this invention (in which fuzzy logicis employed), each fuzzy union label may be computed: (i) with aLukasiewicz fuzzy union operator U(a,b)=min(1,a+b); or (ii) with aprobabilistic fuzzy union operator U(a,b)=a+b−(ab); or (iii) with aZadeh fuzzy union operator U(a,b)=max(a,b); or (iv) with a Yager fuzzyunion operator U(a,b)=min[(1,a^(w)+b^(w))^(1/w)], where w∈(0,∞); or (v)with a Hamacher's sum fuzzy union operator

${{U\left( {a,\ b} \right)} = \frac{a + b - {\left( {2 - \gamma} \right)ab}}{1 - {\left( {1 - \gamma} \right)ab}}},$where γ≥0; or (vi) with a bounded fuzzy union operatorU(a,b)=min(1,√{square root over (a²+b²)}); or (vii) with a drastic sumfuzzy union operator, where U(a,b) equals a if b=0, b if a=0, and 1 forothers.

Different fuzzy union operators may be desirable, depending on the databeing classified. For instance:

As noted above, the Lukasiewicz fuzzy union operator is the lesser of(i) 1 and (ii) the sum of a and b (recall that the highest possibledegree of membership is 1). Thus, the Lukasiewicz fuzzy union operatormay (in some cases) be desirable where: (a) there is a very littleoverlap between fuzzy sets A and B; (b) membership in fuzzy set A is anindependent indication of a condition sought to be diagnosed; and (c)membership in fuzzy set B is also an independent indication of thecondition sought to be diagnosed.

As can be seen from the description above, the probabilistic fuzzy unionoperator is similar to a conventional union operator. Thus, theprobabilistic fuzzy union operator, too, may (in some cases) bedesirable where: (a) there is a very little overlap between fuzzy sets Aand B; (b) membership in fuzzy set A is an independent indication of acondition sought to be diagnosed; and (c) membership in fuzzy set B isalso an independent indication of the condition sought to be diagnosed.

As noted above, the drastic sum fuzzy union operator is 1 if either a orb is non-zero. Thus, the drastic sum fuzzy union operator may (in somecases) be desirable where: (a) even a slight degree of membership infuzzy set A indicates that a condition sought to be diagnosed is fullypresent; and (b) even a slight degree of membership in fuzzy set Bindicates that the condition sought to be diagnosed is fully present.

In each implementation that involves a “union”, the union may instead bea fuzzy union. In each implementation that involves a union operator,the union operator may instead be a fuzzy union operator.

In illustrative implementations of this invention, a CNN (convolutionalneural network) may be trained on a set of images labeled with unionlabels. For example, the CNN that is trained (on a set of images labeledwith union labels) may comprise a LeNet, AlexNet, VGG-16 (VisualGeometry Group 16-layer neural network), VGG-19 (Visual Geometry Group19-layer neural network), ResNet, GoogleNet (e.g., Inception 3),multi-stream CNN, multi-stream multi-channel CNN, FCN (fullyconvolutional network), or U-Net FCN. The CNN, once trained, may be usedto classify a previously unknown image of the same type as the trainingimages. For example, if the training images comprise white-light imageswhere each pixel, respectively, is labeled with a union label, then theCNN, once trained, may classify a previously unknown white-light image.

In some implementations, employing an FCN may provide a significantdecrease in both training and testing time at the expense of requiringmore training data.

Alternatively, in illustrative implementations, any other method ofsupervised machine learning may be employed when training on imageslabeled with union labels. For example, a RNN (recurrent neuralnetwork), RNN with LSTM (long short term memory), RNN with GatedRecurrent Unit, MLP (multi-layered perceptron), ANN (artificial neuralnetwork), or SVM (support vector machine) may be trained on imageslabeled with union labels. Again, once the algorithm is trained, it maybe used to classify a previously unknown image of the same type as thetraining images. Again, for example, if the training images comprisewhite-light images where each pixel, respectively, is labeled with aunion label, then the algorithm, once trained, may classify a previouslyunknown white-light image.

Depending on the particular type of data being classified in aparticular implementation of this invention, the type of machinelearning employed may vary. A method of machine learning well-suited forthe type of data being analyzed may be employed to classify the data.

As noted above, in some implementations, the union labels are generatedbased on a first set of images that have been classified by a computervision algorithm and a second set of images that have been annotated bya human expert. For instance, the first set of images may comprisebiomarker images that are classified by a computer vision algorithmapplying labels (e.g. computer vision generated labels 207) to thebiomarker images.

As used in the following three paragraphs: (a) “initial machineclassifier” means a classifier used to assign computer-generated labelsthat are taken as an input when calculating the union labels; and (b)“union-label-trained classifier” means a classifier that is trained onimages labeled with union labels.

In many cases, the initial machine classifier is a simpler classifierthan the union-label-trained classifier. For example, the machineclassifier may comprise a histogram thresholding operation that assignsmachine labels to pixels with values in specific color ranges when theinput images encode relevant information as specific colors, as somebiomarker images do.

In some cases, both the initial machine classifier andunion-label-trained classifier are the same type of neural network. Forexample, in some cases, both the initial machine classifier and theunion-label-trained classifier comprise a VGG16 neural network. Also,for example, in some cases both the initial machine classifier andunion-label-trained classifier comprise a FCN (fully convolutionalnetwork).

The initial machine classifier may be different than theunion-label-trained classifier (including differences in type,architecture or hyperparameters). For example, in some cases, theinitial machine classifier comprises a neural network that is of adifferent type than (or that has a different architecture than, or thathas different hyperparameters than) the union-label-trained classifier.For example, in some cases, the initial machine classifier is a VGG16network and the union-label-trained classifier is an FCN.

In some cases, the initial machine classifier performs unsupervisedmachine learning. For example, in some cases, the initial machineclassifier comprises AEs (auto-encoders), deep AEs, SAES (stackedauto-encoders), VAE (variational auto-encoder), GAN (generativeadversarial network), or RBMs (Restricted Boltzman Machines), orcomprises a DBN (Deep Belief Network), or performs PCA (principalcomponent analysis) or clustering.

Imaging Hardware

FIG. 8 shows hardware of an imaging system that is configured to acquireand classify an image. In the example shown in FIG. 8: (a) imagingdevice A 801 captures modality A images of object 810; and (b) imagingdevice B 802 captures modality B images of object 810. For instance,imaging device A 801 and imaging device B 802 may comprise white lightcamera 201 and biomarker camera 204, respectively (as shown in FIGS. 2and 3). Or, for example, imaging device A 801 and imaging device B 802may comprise an MRI imaging device and a PET imaging device,respectively. Or, for instance, imaging device A 801 and imaging deviceB 802 may comprise an CT imaging device and a PET imaging device,respectively.

In FIG. 8, a human expert may interact with one or more I/O(input/output) devices 805 to apply expert labels to an image on apixel-by-pixel or patch-by-patch basis. For instance, the human expertmay employ one or more I/O devices 805 to label each pixel of a whitelight image of teeth as either “plaque present” or “plaque not present”.The one or more I/O devices 805 may comprise one or more electronicdisplay screens, computer monitor screens, touch screens, projectors,projector screens, keyboards, computer mice, microphones, speakers,electronic styluses, or haptic transducers. The one or more I/O devices805 may present a graphical user interface (GUI) whereby: (a) a humanuser may input instructions or information or (b) information orinstructions may be displayed to (or otherwise outputted to) a humanuser. For instance, a human expert may employ a “digital brush” in a GUIto apply labels to pixels (e.g., to label pixels as either “plaquepresent” or “plaque not present”).

In FIG. 8, computer 803 may perform a computer vision algorithm toinitially classify an image on a pixel-by-pixel basis or on anpatch-by-patch basis. Computer 803 may perform registration andthresholding, may compute union labels, may apply union labels to animage to create a union-labeled training image, may train a classifieron union-labeled training images, and may, after the classifier istrained, classify previously unseen images. Computer 803 may cause datato be stored in, and to be accessed from, memory device 804.

Prototype

The following 21 paragraphs describe a prototype of this invention.

In this prototype, plaque's presence on part of a tooth is detectedbased on information contained in the immediate neighborhood. Hence, theCNN architecture takes as input an n×n white light image patch andoutputs a prediction of whether the patch's center pixel corresponds toplaque or not. This local patch-based method is desirable forclassifying plaque because plaque's free-form shape makes bounding boxesa poor model of plaque presence; thus, a per-pixel annotation isadvantageous. Training on patches rather than full images additionallyallows for much more training data from fewer images.

In this prototype, because a CNN is trained on the union labels, the CNNlearns more comprehensive signatures of plaque, encompassing bothbiomarker locations and expert annotations which do not have significantoverlap.

In this prototype, the CNN architecture is a truncated version of VGG16.The truncated version is used in order to achieve a small model with agreat enough capacity to learn the training distribution. The inventorsexperimentally determined a depth that did not result in underfitting onthe training set, which is the first thirteen layers of the VGG16architecture, followed by a smaller fully-connected layer of 256 nodesand the final softmax function.

In this prototype, the convolutional neural network architecture takesas input an n×n pixel patch from a standard photograph and outputs aprobability of whether the patch's center pixel corresponds to thecondition or not. In this prototype, the network architecture comprisesthe first through thirteenth layers of VGG16 followed by a smallerfully-connected layer of 256 nodes and a final softmax function. In thisprototype, the CNN was trained using adaptive stochastic gradientdescent with momentum, and the loss function captures the softmaxcross-entropy in classification of all patches in the currentmini-batch. In this prototype, gradients are calculated bybackpropagation. To help prevent overfitting to the training data, thetraining is performed with a dropout probability of 0.5.

In this prototype, a fully-trained convolutional neural networkprocesses all 21×21 patches of a previously unseen photograph andproduces a classification for each pixel corresponding to the predictedpresence or absence of the clinical condition. The information fromspecialized medical images and expert annotations is incorporated duringthe training process; the trained convolutional neural network mayclassify a standard white-light photograph.

In this prototype, there is a trade-off between patch size and theamount of different images required; larger patches contain morecontextual information around the center pixel but can therefore eachcapture less of the variation than a smaller patch would, therebyrequiring more training patches. That is, the space of variation islarger for larger patches. After initial optimization experiments, apatch size of 21×21 pixels was chosen. Two classifiers are implementedusing TensorFlow™ software, and training is performed on an NVIDIACorporation GM200 GeForce GTX TITAN X. Training hyperparameters weredetermined through grid search: mini-batch size of 100, learning rate of1×10⁻⁶, 3 epochs.

In this prototype, two different versions of the imaging system weretested.

In this prototype, the first version of the imaging system comprises anActeon® Soprocare® intraoral probe (“CD probe”) that operates in (amongother things): (a) a white light mode and (b) a “plaque” mode. In whitelight mode, the CD probe illuminates teeth with white light and captureswhite light images. In the plaque mode, the CD probe illuminates teethwith both 450 nm and white light, and then digitally embellishes thecolor of newly-formed plaque-affected areas in hues of yellow andorange.

In this prototype, the second version of the imaging system comprises anintraoral probe (the “RD probe”) that operates in two different modes.In the first mode, the RD probe emits white light and captures whitelight images. In the second mode, the RD probe illuminates plaque with405 nm light and captures red fluorescent light that (a) is emitted byporphyrin associated with mature plaque biofilms formed by anaerobicbacteria and (b) is filtered to remove blue and violet light.

In this prototype, two classifiers successfully predict the location ofdental plaque in white light dental images with high accuracy. In thisprototype, there are two fully trained and validated CNNs. Each of theseCNNS (after learning from both fluorescent biomarker images as well asexpert labels), accepts standard white light intraoral images as inputsand predicts the location of plaque pixels with high sensitivity andspecificity.

In a test of this prototype, 27 adult subjects consented to imaging ofincisors and canines. Each subject was imaged sequentially in (1) whitelight mode, (2) CD plaque mode, (3) RD white light mode, and (4) RDplaque mode. In total, 47 pairs of images were captured with the CD, and49 pairs were captured with the RD. Illumination conditions were kept asconstant as possible across subjects in each dataset. Each pair ofimages: (a) comprised a white light image captured by the CD probe and afluorescent biomarker image captured by the CD probe; or (b) comprised awhite light image captured by the RD probe and a fluorescent biomarkerimage captured by the RD probe

In this test of the prototype, datasets from the CD probe and RD probecomprise white light images and corresponding fluorescent biomarkerimages. The white light and biomarker images were registered with eachother. To perform the registration, a perspective transformation thatminimizes the mean-squared error between the intensities of the imageswas applied to the biomarker images.

In this test of the prototype, a computer computed a binary pixel-levelclassification of plaque and not-plaque of the biomarker image. Thispixel-by-pixel classification was by histogram thresholding thefluorescent biomarker images using empirically-determined thresholds foreach dataset. The devices did not capture all plaque in an image due tothe absence of porphyrins in some plaque.

In this test of the prototype, expert dental professionals independentlyannotated regions showing plaque on the white light images of teethcaptured by the CD probe and the RD probe.

In this test of the prototype, union labels were calculated. In thistest o the prototype, the union labels represent the full extent ofplaque detected by human experts, by computer vision labeling offluorescent biomarker images, or by both. In this test of the prototype,if either or both indicate that plaque is present, the union labelindicates that plaque is present. This is desirable approach because thefluorescent biomarker images do not show all plaque (because porphyrinsare not present in some plaque) and because the human experts do notdetect all plaque in white light images. In this test of the prototype,only a small percentage of the per-pixel plaque labels in each unionlabel image were detected by both the expert and device, indicatingdistinct roles for each labeling method.

In this test of the prototype, two CNN were trained. One CNN was trainedon pairs of images captured by the CD probe. The other CNN was trainedon pairs of images captured by the RD probe.

In this test of the prototype, these pixel-level annotations containedin the union are the labels used to train and test the CNN classifiers.

In this test of the prototype, the pixel dimensions of the RD imageswere 512×384 while those of the CD images were 640×480. Accounting forthe margins, each type of image were represented by 183,393 or 290,625patches, respectively, where each patch consisted of 21×21 pixels. Arandom sample of half the patches in each training image were used fortraining to limit overfitting on extremely similar patches, while allpatches in test images were used for testing. Patches from a singleimage were not split among the train and test sets.

In this test of the prototype, each image was assigned to one of threegroups based on the amount of plaque in the union label: low plaque,medium plaque, high plaque. Then 70% of images from each group wererandomly assigned to the training set and the remaining 30% to the testset to ensure that the plaque quantity distribution of the training andtest sets were roughly the same. For the CD dataset, 4,687,980 patchesfrom 33 images were used for training and 3,977,694 patches from 14images were used for testing. For the RD dataset, 3,209,360 patches from35 images were used for training and 2,750,895 patches from 14 imageswere used for testing. Images were normalized in the RGB colorspace, andwhite light images in the RD dataset all received the same colorbalancing to account for variation in illumination. Each feature in thetraining set was standardized to have a zero mean and unit variance, andthe transformations with the same parameters were applied to eachfeature in the test set.

In this test of this prototype, training accuracy and test accuracy were87.93% and 84.67%, respectively, for the RD dataset.

In this test of this prototype, training accuracy and test accuracy were80.83% and 87.18%, respectively, for the CD dataset.

The prototype described in the preceding 21 paragraphs is a non-limitingexample of this invention. This invention may be implemented in manyother ways.

Sensors

This invention is not limited to classifying images.

In some implementations, a classifier (which has been trained on unionlabels) classifies measurements taken by sensors. As a non-limitingexample: (a) a first set of sensor measurements may be taken by athermometer and a second set of sensor measurements may be taken by avoltmeter; and (b) the first and second sets of sensor measurements maybe employed to create union labels.

In some implementations: (a) a classifier (which has been trained onunion labels) classifies sensor data taken by more than two types ofsensors; and (b) more than two types of sensors take measurements thatare used to create the union labels. For instance, two, three, four,five, six or more types of sensors may be employed.

In some implementations, at least one set of data (e.g., a set of sensordata initially classified by machine or initially annotated by a human)comprises sensor data regarding physiologic phenomenon, such as sensordata regarding heart rate, respiration rate, blood pressure, bloodchemistry, EDA (electro-dermal activity), or such as EEG(electroencephalography) data, other electrophysiological data, orultrasound data. Also, for example, in some implementations, at leastone set of data (e.g., a set of sensor data initially classified bymachine or initially annotated by a human) comprises weather data orseismographic data. Also, for example, in some implementations, at leastone set of data (e.g., a set of sensor data initially classified bymachine or initially annotated by a human) comprises sensor dataregarding pressure, temperature, contact, proximity, movement,acceleration, sound, vibration, electromagnetic radiation, electricalphenomenon (e.g., resistance, capacitance, inductance, impedance,current), or magnetic phenomenon, gravity, weight, or material property.Also, for example, in some implementations, at least one set of data(e.g., a set of sensor data initially classified by machine or initiallyannotated by a human) comprises sensor data regarding a phenomenon thatvaries as a function of spatial position, or that varies as a functionof time, or that varies as a function of any one or more physicalparameters or other parameters.

In some implementations of this invention: (a) X different types ofsensors are employed, where X≥2; (b) registration is performed in such away that, after registration, the sensor measurements taken by the Xtypes of sensors comprise groups of registered measurements where all ofthe measurements in each particular group are registered with each otherand are measurements of the same physical object or same physical event;(c) each group of registered measurements includes X measurements, oneby each type of sensor; and (d) for each specific datapoint in a groupof registered measurements, the union label is the union of the initiallabels for that specific datapoint. For instance, in some cases, if agroup of registered measurements includes measurements taken by onlythree types of sensors and the initial labels for a specific datapointare “1” for the first type of sensor, “0” for the second type of sensor,and “0” for the third type of sensor, then the union label for thatspecific datapoint is “1”. Likewise, in some cases, if a group ofregistered measurements includes measurements taken by only three typesof sensors and the initial labels for a specific datapoint are “0” forthe first type of sensor, “0” for the second type of sensor, and “0” forthe third type of sensor, then the union label for that specificdatapoint is “0”. In some cases: (a) sensor measurements taken by atleast one of the X types of sensor are initially labeled by a humanexpert; and (b) sensor measurements taken by at least one other of the Xtypes of sensors are initially labeled by a computer algorithm (e.g., asupervised or unsupervised machine learning algorithm).

FIG. 9 shows a flowchart for a method that employs union-labeled sensordata to train a classifier, in such a way that the trained classifiermay accurately classify previously unseen sensor data.

In the example shown in FIG. 9, sensor A 901 differs, in structure andfunction, from sensor B 904. Sensor A 901 takes measurements 902 ofphenomenon 900. Labels 903 are applied to the sensor A measurements, ona measurement-by-measurement basis. In FIG. 9, sensor B 904 takesmeasurements 905 of phenomenon 900. Labels 907 may be applied to thesensor A measurements, on a measurement-by-measurement basis. Each label(out of labels 903 and 907) may classify a measurement as either “X” or“not X”.

In some cases, labels 903 are created by a human expert and labels 907are generated by a computer algorithm. In some cases, labels 903 aregenerated by a computer algorithm and labels 907 are created by a humanexpert. In some cases, labels 903 and 907 are all created by one or morehuman experts. In some cases, labels 903 and 907 are all generated by acomputer algorithm.

In FIG. 9, registration 908 is performed to register the sensor Ameasurements 902 and sensor B measurements 905. A computer calculatesthe union 909 of the expert labels and the union labels on ameasurement-by-measurement basis, so that there is a union label foreach measurement. The union labels are applied to the sensor Ameasurements 902 to create union-labeled sensor A measurements 910.

This process may be repeated for multiple phenomena, to acquire atraining dataset that comprises union-labeled measurements. Training 911may be performed to train a classifier on a set of training data, whereeach measurement in the training data is a union-labeled sensor Ameasurement. Then, the trained classifier 912 may classify previouslyunseen sensor A measurements 920 by predicting union labels 921 for thepreviously unseen sensor A measurements, on a measurement-by-measurementbasis.

In some implementations, a classifier (which has been trained on unionlabels) classifies regions of sensor readings. For instance, each regionbeing classified may comprise a single datapoint (single measurement bya sensor). Alternatively, each region being classified may comprise agroup of measurements (e.g., that are captured during a specific timeperiod or that are captured for a specific spatial region).

FIG. 10 shows hardware of a sensor system that is configured to acquireand classify sensor data. In the example shown in FIG. 10, a humanexpert may interact with one or more I/O (input/output) devices 805 toapply expert labels to sensor measurements on ameasurement-by-measurement basis or region-by-region basis (where eachregion is a group of measurements).

In FIG. 10, computer 803 may perform a computer algorithm to initiallyclassify sensor measurements on a measurement-by-measurement orregion-by-region basis (where each region is a group of measurements).Computer 803 may perform registration and thresholding, may computeunion labels, may apply union labels to a sensor measurement to create aunion-labeled measurement, may train a classifier on union-labeledmeasurements, and may, after the classifier is trained, classifypreviously unseen sensor measurements.

Data Sets

This invention is not limited to classifying data captured by sensors.

In some implementations, a classifier (which has been trained on unionlabels) classifies any type of data. For example, in some cases, thedata being labeled may comprise economic data, financial data (such asdata regarding stock prices or prices of other financial assets), ordata regarding education, music, politics or other human activity.

In some implementations of this invention: (a) X different types of dataare employed, where X≥2; (b) registration is performed to register the Xtypes of data; and (c) for each specific datapoint, the union label isthe union of the initial labels for that datapoint. For instance, insome cases, if the registered data includes only three types of data andthe initial labels for a specific datapoint are “1” for the first typeof data, “0” for the second type of data, and “0” for the third type ofdata, then the union label for that specific datapoint is “1”. Likewise,in some cases, if the registered data includes only three types of dataand the initial labels for a specific datapoint are “0” for the firsttype of data, “0” for the second type of data, and “0” for the thirdtype of data, then the union label for that specific datapoint is “0”.In some cases: (a) at least one of the X types of data is initiallylabeled by a human expert; and (b) at least one other of the X types ofdata is initially labeled by a computer algorithm (e.g., a supervised orunsupervised machine learning algorithm).

FIG. 11 shows a flowchart for a method that employs union-labeled datato train a classifier, in such a way that the trained classifier mayaccurately classify previously unseen data.

In the example shown in FIG. 11, dataset A 1102 is a different type ofdata than dataset B 1104. For instance: (a) dataset A 1102 may comprisetranscripts of broadcasters' words during broadcast soccer games and maybe initially labeled by a human expert; and (b) dataset B 1104 maycomprise audio recordings of sounds made by audience in the stadiumduring the same soccer games and may be initially labeled by soundrecognition software. Labels 1103 may be applied to dataset A 1102.Labels 1107 may be applied to dataset B 1104. Each label (out of labels1103 and 1107) may classify a datapoint as either “X” or “not X”.

In some cases, labels 1103 are created by a human expert and labels 1107are generated by a computer algorithm. In some cases, labels 1103 aregenerated by a computer algorithm and labels 1107 are created by a humanexpert. In some cases, labels 1103 and 1107 are all created by one ormore human experts. In some cases, labels 1103 and 1107 are allgenerated by a computer algorithm.

In FIG. 9, registration 1108 is performed to register dataset sensor A1102 and dataset B 1105. A computer may calculate the union 1109 of theexpert labels and the union labels on a datapoint-by-datapoint basis, sothat there is a union label for each datapoint. The union labels may beapplied to dataset A 1102 to create a union-labeled dataset A 1110.

This process may be repeated, to acquire a training dataset. Training1111 may be performed to train on this training dataset. Then, thetrained classifier 1112 may classify a previously unseen dataset 1120(of the same type as dataset A) by predicting union labels 1121 for thepreviously unseen dataset.

Registration

Registration of Images: As noted above, images may be registered beforecomputing union labels. In some cases, registration is performed bycomparing intensity patterns by correlation metrics. For instance,registration may be performed by a perspective transformation thatminimizes the mean-squared error between the intensities of the images.In some cases, registration is performed by detecting correspondingfeatures in images. For instance, the corresponding features may bepoints (e.g., such as CCPs (corresponding control points)), lines orcontours. Registration may involve performing rotation, scaling,shearing or other affine transformations of an image being registered.In some cases, registration involves warping of local regions of animage being registered. The registration methods described above in thisparagraph may also be applied to registering sensor data or other data,with appropriate modification if needed. For instance, in some cases, ifsensor data (or other data) is being registered and if that sensor data(or other data) does not have units of intensity, then registration maybe performed by comparing patterns of magnitude (or size, amount oramplitude) of a phenomenon represented by the sensor data (or by otherdata).

Registration of Sensor Data: In some implementations, sensormeasurements: (a) are taken by different types of sensors; (b) are eacha function of the same independent variable; and (c) may be registeredby causing sensor measurements for the same value of that independentvariable to be paired with each other, grouped with each other, orotherwise aligned with each other. As a non-limiting example ofregistration, consider two sets of sensor measurements f(t) and g(t),where sensor measurements f(t) are taken by a first type of sensor,where sensor measurements g(t) are taken by a different type of sensor,and where sensor measurements f (t) and g(t) both vary as a function ofthe same independent variable t. For instance, t may be time. In thisexample, to say that sensor measurements f(t) and g(t) are “registered”with each other means that, for each specific value t_(s) in a set ofvalues of t, sensor measurements f(t_(s)) and g(t_(s)) are paired witheach other, grouped with each other, or otherwise aligned with eachother. As another non-limiting example of registration, consider twosets of sensor measurements f(x,y) and g(x,y), where sensor measurementsf(x,y) are taken by a first type of sensor, where sensor measurementsg(x,y) are taken by a different type of sensor, and where sensormeasurements f(x,y) and g(x,y) both vary as a function of the sameindependent variables x and y. For instance, x and y may be Euclideanspatial coordinates. In this example, to say that sensor measurementsf(x,y) and g(x,y) are “registered” with each other means that, for eachspecific point (x_(s),y_(s)) in a set of (x,y) points, sensormeasurements f(x_(s),y_(s)) and g(x_(s),y_(s)) are paired with eachother, grouped with each other, or otherwise aligned with each other.

Registration of data: In some implementations, different types of data:(a) are each a function of the same independent variable; and (b) may beregistered by causing datapoints for the same value of that independentvariable to be paired with each other, grouped with each other, orotherwise aligned with each other. As a non-limiting example ofregistration, consider two datasets f(t) and g(t), where dataset f(t)and dataset g(t) both vary as a function of the same independentvariable t. In this example, to say that dataset f(t) and dataset g(t)are “registered” with each other means that, for each specific valuet_(s) in a set of values of t, datapoint f(t_(s)) and datapoint g(t_(s))are paired with each other, grouped with each other, or otherwisealigned with each other. As another non-limiting example ofregistration, consider two datasets f(x,y) and g(x,y), where datasetf(x,y) and dataset g(x,y) both vary as a function of the sameindependent variables x and y. In this example, to say that datasetf(x,y) and dataset g(x,y) are “registered” with each other means that,for each specific point (x_(s),y_(s)) in a set of (x,y) points,datapoint f (x_(s),y_(s)) and datapoint g(x_(s),y_(s)) are paired witheach other, grouped with each other, or otherwise aligned with eachother.

Single Modality

In some implementations of this invention, only one imaging modality (oronly one sensing modality) is employed.

For instance, in some cases, two sets of images are captured by the samecamera. The first set of images may be initially labeled by a humanexpert, on a pixel-by-pixel basis. The second set of images may beinitially labeled by computer vision, on a pixel-by-pixel basis. Thefirst and second sets of images may be registered. Union labels may becalculated. For each pixel of the registered images, the union label maybe the union of the expert label and computer vision label for thatpixel. This process may be repeated, to create a training set ofunion-labeled images. A classifier may be trained on this training set.Once trained, the classifier may classify a previously unseen image(which was captured by the same camera) by predicting union labels forthat image.

Likewise, in some cases, two sets of measurements are taken by the samesensor. The first set of measurements may be initially labeled by ahuman expert. The second set of measurements may be initially labeled bya computer algorithm. The first and second sets of measurements may beregistered. Union labels may be calculated. For each measurement, theunion label may be the union of the expert label and computer visionlabel for that measurement. This process may be repeated, to create atraining set of union-labeled measurements. A classifier may be trainedon this training set. Once trained, the classifier may classify apreviously unseen measurement (which was captured by the same sensor) bypredicting union labels for that measurement.

Projecting Labels

This invention is not limited to union labels.

In some implementations, instead of calculating union labels, labelsfrom a first image (which was captured by a first imaging modality) areprojected onto a second, unlabeled image (which was captured by a secondimaging modality) to create a projection-labeled image. Thisprojection-labeled image may comprise a training image. The processdescribed in the preceding sentence may be repeated to create a set oftraining images, where each training image comprises aprojection-labeled image (which was initially captured by the secondimaging modality and then labeled with projected labels). For instance,each training image in the set of training images may be an image of adifferent sample (e.g., a different sample of the same type of tissue).A classifier may then be trained these projection-labeled images. Oncetrained, the classifier may classify a previously unseen image (whichwas captured by the second imaging modality) by predicting projectionlabels for that image. In these alternative implementations, the firstand second images may be registered. Then, after this registration,labels from the first image may be projected onto a second, unlabeledimage by, for each particular pixel in the first image, applying thelabel for the particular pixel to a corresponding pixel in the secondimage. The resulting labels on the second (previously unlabeled) imagemay comprise projection labels. Alternatively, the projection of labelsmay be on a patch-by-patch basis, instead of a pixel-by-pixel basis.

Each description herein of a method that employs union labels mayinstead be implemented with projection labels, by making appropriatemodifications.

For example, in FIG. 2: (a) the expert white light labels 203 may beomitted, (b) the arrows from 202 to 203 and from 203 to 208 may bereplaced by an arrow from 202 to 208 (which may join with the arrow from207 to 208); (c) the word “union” in 209 may be replaced by the phrase“projecting labels”; (d) the phrase “union-labeled” in 210 may bereplaced by the phrase “projection-labeled”; and (e) the phrase “unionlabels” in 221 may be replaced by the phrase “predicted projectionlabels”.

Likewise, in FIG. 3: (a) the expert white light labels 203 may beomitted, (b) the arrows from 202 to 203 and from 203 to 208 may bereplaced by an arrow from 202 to 208 (which may join with the arrow from207 to 208); (c) the word “union” in 209 may be replaced by the phrase“projecting labels”; (d) the phrase “union-labeled” in 310 may bereplaced by the phrase “projection-labeled”; and (e) the phrase “unionlabels” in 321 may be replaced by the phrase “predicted projectionlabels”.

Likewise, in FIG. 4: (a) the modality A labels 403 may be omitted, (b)the arrows from 402 to 403 and from 403 to 408 may be replaced by anarrow from 402 to 408 (which may join with the arrow from 407 to 408);(c) the word “union” in 409 may be replaced by the phrase “projectinglabels”; (d) the phrase “union-labeled” in 410 may be replaced by thephrase “projection-labeled”; and (e) the phrase “union labels” in 421may be replaced by the phrase “predicted projection labels”.

Likewise, in FIG. 9: (a) the sensor A labels 903 may be omitted, (b) thearrows from 902 to 903 and from 903 to 908 may be replaced by an arrowfrom 902 to 908 (which may join with the arrow from 907 to 908); (c) theword “union” in 909 may be replaced by the phrase “projecting labels”;(d) the phrase “union-labeled” in 910 may be replaced by the phrase“projection-labeled”; and (e) the phrase “union labels” in 921 may bereplaced by the phrase “predicted projection labels”.

Likewise, in FIG. 11: (a) the dataset A labels 1103 may be omitted, (b)the arrows from 1102 to 1103 and from 1103 to 1108 may be replaced by anarrow from 1102 to 1108 (which may join with the arrow from 1107 to1108); (c) the word “union” in 1109 may be replaced by the phrase“projecting labels”; (d) the phrase “union-labeled” in 1110 may bereplaced by the phrase “projection-labeled”; and (e) the phrase “unionlabels” in 1121 may be replaced by the phrase “predicted projectionlabels”.

Non-Transitory Media

In some implementations, a non-transitory machine-accessible medium(e.g., a compact disk or thumb drive) has instructions encoded thereonfor enabling one or more computers to perform one or more of theComputer Tasks (as defined herein).

Downloading Software

In some implementations, this invention comprises participating in adownload of software, either as a computer providing the software or asa computer receiving the software, wherein the software comprisesinstructions for enabling one or more computers to perform one or moreof the Computer Tasks.

Software

In the Computer Program Listing above, six computer program files arelisted. These six computer program files comprise software employed in aprototype implementation of this invention. To run three of these(create_patches_random_train.txt, create_patches_test.txt andvgg_whitelight_cnn.txt) as Python™ software, the filename extension foreach would be changed from “.txt” to “.py”. To run the other three ofthese (interpretResults.txt, makeROC.txt and thresholdROCs.txt) asMatlab® software, the filename extension for each would be changed from“.txt” to “.m”. In addition, before running these files, the filenamesmay be revised by replacing each underscore “_” in the filename with adash “-” (e.g., change “create_patches_random_train.txt” to“create-patches-random-train.py”.

Here is a description of these six computer program files:

(1) create_patches_random_train.txt extracts 21×21-pixel patches fromtraining images, randomly sub/oversampling to get a desired class ratio;

(2) create_patches_test.txt extracts all 21×21-pixel patches from eachimage in the test set;

(3) interpretResults.txt constructs ROC (receiver operatingcharacteristic) curve and precision-recall curves from .csv files (andconverts from one-hot labels to indices);

(4) makeROC.txt generates ROC;

(5) thresholdROCs.txt averages ROC; and

(6) vgg_whitelight_cnn.txt trains and tests a convolutional neuralnetwork for classifying the center pixel of an image patch from a whitelight dental image as plaque or not-plaque.

This invention is not limited to the software set forth in these sixcomputer program files. Other software may be employed. Depending on theparticular implementation, the software used in this invention may vary.

Computers

In illustrative implementations of this invention, one or more computers(e.g., servers, network hosts, client computers, integrated circuits,microcontrollers, controllers, field-programmable-gate arrays, personalcomputers, digital computers, driver circuits, or analog computers) areprogrammed or specially adapted to perform one or more of the followingtasks: (1) to control the operation of, or interface with, hardwarecomponents of an imaging system, including any light source, radiationsource, light detector, radiation detector, or actuator (e.g., anelectric motor for actuating a scan, such as a tomographic scan); (2) toperform a computer algorithm to initially label a set of data (e.g., toinitially label an image on a pixel-by-pixel basis or a patch-by-patchbasis, or to initially label sensor measurements, or to initially labelother datapoints); (3) to control the operation of, and to receive datafrom, one or more input/output devices; (4) to control a graphical userinterface whereby a human expert inputs labels (e.g., on apixel-by-pixel basis or a patch-by-patch basis); (5) to calculate unionlabels; (6) to train a classifier (e.g., a neural network) withunion-labeled data (e.g. to train with union-labeled images, or withunion-labeled sensor data, or with other union-labeled data); (7) toemploy a trained classifier, which has been trained with union-labeleddata, to perform classification; (8) to project labels from a labeledimage (or labeled sensor measurements or other labeled dataset) onto anunlabeled image (or unlabeled sensor measurements or other unlabeleddataset); (9) to train a classifier (e.g., a neural network) withprojection-labeled data (e.g. to train with projection-labeled images,or with projection-labeled sensor data, or with other projection-labeleddata); (10) to employ a trained classifier, which has been trained withprojection-labeled data, to perform classification; (11) to analyze animage to detect dental plaque or gingivitis; (12) to receive data from,control, or interface with one or more sensors; (13) to perform anyother calculation, computation, program, algorithm, or computer functiondescribed or implied herein; (14) to receive signals indicative of humaninput; (15) to output signals for controlling transducers for outputtinginformation in human perceivable format; (16) to process data, toperform computations, and to execute any algorithm or software; and (17)to control the read or write of data to and from memory devices (tasks1-17 of this sentence referred to herein as the “Computer Tasks”). Theone or more computers (e.g. 803) may, in some cases, communicate witheach other or with other devices: (a) wirelessly, (b) by wiredconnection, (c) by fiber-optic link, or (d) by a combination of wired,wireless or fiber optic links.

In exemplary implementations, one or more computers are programmed toperform any and all calculations, computations, programs, algorithms,computer functions and computer tasks described or implied herein. Forexample, in some cases: (a) a machine-accessible medium has instructionsencoded thereon that specify steps in a software program; and (b) thecomputer accesses the instructions encoded on the machine-accessiblemedium, in order to determine steps to execute in the program. Inexemplary implementations, the machine-accessible medium may comprise atangible non-transitory medium. In some cases, the machine-accessiblemedium comprises (a) a memory unit or (b) an auxiliary memory storagedevice. For example, in some cases, a control unit in a computer fetchesthe instructions from memory.

In illustrative implementations, one or more computers execute programsaccording to instructions encoded in one or more tangible,non-transitory, computer-readable media. For example, in some cases,these instructions comprise instructions for a computer to perform anycalculation, computation, program, algorithm, or computer functiondescribed or implied herein. For example, in some cases, instructionsencoded in a tangible, non-transitory, computer-accessible mediumcomprise instructions for a computer to perform the Computer Tasks.

Network Communication

In illustrative implementations of this invention, electronic devices(e.g., 801, 802, 803, 805, 901, 902) are configured for wireless orwired communication with other devices in a network.

For example, in some cases, one or more of these electronic devices eachinclude a wireless module for wireless communication with other devicesin a network. Each wireless module may include (a) one or more antennas,(b) one or more wireless transceivers, transmitters or receivers, and(c) signal processing circuitry. Each wireless module may receive andtransmit data in accordance with one or more wireless standards.

In some cases, one or more of the following hardware components are usedfor network communication: a computer bus, a computer port, networkconnection, network interface device, host adapter, wireless module,wireless card, signal processor, modem, router, cables or wiring.

In some cases, one or more computers (e.g., 803) are programmed forcommunication over a network. For example, in some cases, one or morecomputers are programmed for network communication: (a) in accordancewith the Internet Protocol Suite, or (b) in accordance with any otherindustry standard for communication, including any USB standard,ethernet standard (e.g., IEEE 802.3), token ring standard (e.g., IEEE802.5), wireless standard (including IEEE 802.11 (wi-fi), IEEE 802.15(bluetooth/zigbee), IEEE 802.16, IEEE 802.20 and including any mobilephone standard, including GSM (global system for mobile communications),UMTS (universal mobile telecommunication system), CDMA (code divisionmultiple access, including IS-95, IS-2000, and WCDMA), or LTS (long termevolution)), or other IEEE communication standard.

Definitions

The terms “a” and “an”, when modifying a noun, do not imply that onlyone of the noun exists. For example, a statement that “an apple ishanging from a branch”: (i) does not imply that only one apple ishanging from the branch; (ii) is true if one apple is hanging from thebranch; and (iii) is true if multiple apples are hanging from thebranch.

To compute “based on” specified data means to perform a computation thattakes the specified data as an input.

Non-limiting examples of a “camera” include: (a) a digital camera; (b) adigital grayscale camera; (c) a digital color camera; (d) a videocamera; (e) a light sensor or image sensor, (f) a set or array of lightsensors or image sensors; (g) an imaging system; (h) a light fieldcamera or plenoptic camera; (i) a time-of-flight camera; and (j) a depthcamera. In some cases, a camera includes any computers or circuits thatprocess data captured by the camera.

The term “comprise” (and grammatical variations thereof) shall beconstrued as if followed by “without limitation”. If A comprises B, thenA includes B and may include other things.

Predicting labels is a non-limiting exampling of “classifying”.Generating labels is a non-limiting example of “classifying”.

Non-limiting examples of a “classifier” include: (a) a neural network;(b) a convolutional neural network; and (c) a machine learning modelthat performs classification.

The term “computer” includes any computational device that performslogical and arithmetic operations. For example, in some cases, a“computer” comprises an electronic computational device, such as anintegrated circuit, a microprocessor, a mobile computing device, alaptop computer, a tablet computer, a personal computer, or a mainframecomputer. In some cases, a “computer” comprises: (a) a centralprocessing unit, (b) an ALU (arithmetic logic unit), (c) a memory unit,and (d) a control unit that controls actions of other components of thecomputer so that encoded steps of a program are executed in a sequence.In some cases, a “computer” also includes peripheral units including anauxiliary memory storage device (e.g., a disk drive or flash memory), orincludes signal processing circuitry. However, a human is not a“computer”, as that term is used herein.

“Computed tomography” or “CT” means x-ray computed tomography.

A non-limiting example of “computer vision” is an algorithm thatanalyzes an image and applies labels to the image, on a pixel-by-pixelbasis. Another non-limiting example of “computer vision” is an algorithmthat analyzes an image and applies labels to the image, on apatch-by-patch basis. Another non-limiting example of “computer vision”is an algorithm that analyzes an image and applies a label to the entireimage.

“Computer vision label” or “computer vision generated label” means alabel produced by computer vision.

To say that a first image “corresponds” to a second image means that thefirst and second images are registered with each other. To say that afirst pixel of a first image “corresponds” to a second pixel of a secondimage, which second image is registered with the first image, means thatspatial position of the first pixel in the first image corresponds tospatial position of the second pixel in the second image. To say that afirst patch of a first image “corresponds” to a second patch of a secondimage, which second image is registered with the first image, means thatspatial position of the first patch in the first image corresponds tospatial position of the second patch in the second image.

“Defined Term” means a term or phrase that is set forth in quotationmarks in this Definitions section.

For an event to occur “during” a time period, it is not necessary thatthe event occur throughout the entire time period. For example, an eventthat occurs during only a portion of a given time period occurs “during”the given time period.

The term “e.g.” means for example.

The fact that an “example” or multiple examples of something are givendoes not imply that they are the only instances of that thing. Anexample (or a group of examples) is merely a non-exhaustive andnon-limiting illustration.

“Expert label” means a label inputted by a human expert.

Unless the context clearly indicates otherwise: (1) a phrase thatincludes “a first” thing and “a second” thing does not imply an order ofthe two things (or that there are only two of the things); and (2) sucha phrase is simply a way of identifying the two things, respectively, sothat they each may be referred to later with specificity (e.g., byreferring to “the first” thing and “the second” thing later). Forexample, unless the context clearly indicates otherwise, if an equationhas a first term and a second term, then the equation may (or may not)have more than two terms, and the first term may occur before or afterthe second term in the equation. A phrase that includes a “third” thing,a “fourth” thing and so on shall be construed in like manner.

“Fluorescent image” means an image that is formed by at leastfluorescent light.

“Filtered fluorescent image” means an image that is formed by at leastfluorescent light, which fluorescent light has been filtered.

“For instance” means for example.

“Fuzzy union label” means a label that is a fuzzy union of multipleother labels.

“Fuzzy union-labeled image” means an image to which one or more fuzzyunion labels have been applied.

“Fuzzy union-labeled modality A image” means a modality A image to whichone or more fuzzy union labels have been applied.

“Fuzzy union operator” means an operator that is an element of the setconsisting of (i) the operator defined by U(a,b)=min(1,a+b); (ii) theoperator defined by U(a,b)=a+b−(ab); (iii) the operator defined byU(a,b)=max(a,b); (iv) the operator defined byU(a,b)=min[(1,a^(w)+b^(w))^(1/w)], where w∈(0,∞); (v) the operatordefined by

${{U\left( {a,\ b} \right)} = \frac{a + b - {\left( {2 - \gamma} \right)ab}}{1 - {\left( {1 - \gamma} \right)ab}}},$where γ>0; (vi) the operator defined by U(a,b)=min(1,√{square root over(a²+b²)}); and (vii) the operator defined by U(a,b) equals a if b32 0, bif a=0, and 1 for others. For purposes of the preceding sentence: (i)a∈[0,1] is the degree of membership of x in a fuzzy set A; (ii) b∈[0,1]is the degree of membership of x in a fuzzy set B; and (iii) U(a,b) isthe fuzzy union of a and b. For instance, in the preceding sentence, xmay be a pixel in an image.

To say a “given” X is simply a way of identifying the X, such that the Xmay be referred to later with specificity. To say a “given” X does notcreate any implication regarding X. For example, to say a “given” X doesnot create any implication that X is a gift, assumption, or known fact.

“Herein” means in this document, including text, specification, claims,abstract, and drawings.

As used herein, an image “of X” means an image of at least X.

As used herein: (1) “implementation” means an implementation of thisinvention; (2) “embodiment” means an embodiment of this invention; (3)“case” means an implementation of this invention; and (4) “use scenario”means a use scenario of this invention.

The term “include” (and grammatical variations thereof) shall beconstrued as if followed by “without limitation”.

“Initial label” means a label that is not a union label.

The terms “modality A image” and “modality B image”, when used togetherin the same sentence or sentence fragment, mean a first image and asecond image, respectively, where either: (a) the first image differsfrom the second image in the type of contrast agent, dye, stain,radionuclide, or radioligand employed; or (b) the first image and secondimage are captured by a first imaging device and a second imagingdevice, respectively, the first imaging device being technologicallydifferent than the second imaging device.

The terms “modality A imaging device” and “modality B imaging device”,when used together in the same sentence or sentence fragment, mean afirst device and a second device, respectively, where the first deviceis configured to capture a modality A image and the second device isconfigured to capture a modality B image.

“N-ary classification” means a classification in which there are Npermitted classes, where N≥3.

“MM” means magnetic resonance imaging.

The term “or” is inclusive, not exclusive. For example, A or B is trueif A is true, or B is true, or both A and B are true. Also, for example,a calculation of A or B means a calculation of A, or a calculation of B,or a calculation of A and B.

A parenthesis is simply to make text easier to read, by indicating agrouping of words. A parenthesis does not mean that the parentheticalmaterial is optional or may be ignored.

Unless the context clearly indicates otherwise, “patch” means a patch ofpixels.

“PET” means positron emission tomography.

“Pre-training image” means an image, the labels of which are used tohelp create union labels.

“Previously unseen” image, in the context of a classifier, means animage that: (a) is not a training image on which the classifier hastrained; and (b) is not an image, labels of which have been used to helpcreate union labels that were applied to a training image on which theclassifier has trained.

As used herein, to “project” labels from a first image to a second imagemeans, for each particular region in a set of regions in the firstimage, applying a label for the particular region to a correspondingregion in the second image. For purposes of the preceding sentence, a“region” means a pixel or a patch of pixels.

The terms “sensor A” and “sensor B”, when used together in the samesentence or sentence fragment, mean a first sensor and a second sensor,where the first sensor is technologically different than the secondsensor.

As used herein, a non-limiting example of a “sentence fragment” is anentire patent claim.

As used herein, the term “set” does not include a group with noelements.

Unless the context clearly indicates otherwise, “some” means one ormore.

As used herein, a “subset” of a set consists of less than all of theelements of the set.

The term “such as” means for example.

To say that a first imaging device is “technologically different” than asecond imaging device means that the first imaging device differs, atleast partially in structure and at least partially in function, fromthe second imaging device. To say that a first sensor is“technologically different” than a second sensor means that the firstsensor differs, at least partially in structure and at least partiallyin function, from the second sensor.

To say that a machine-readable medium is “transitory” means that themedium is a transitory signal, such as an electromagnetic wave.

As used herein, a “union label” means the inclusive disjunction (orequivalently, Boolean OR) of multiple other labels. However, thisdefinition of “union label” does not apply to a fuzzy union label.

“Union-labeled image” means an image to which one or more union labelshave been applied.

“Union-labeled modality A image” means a modality A image to which oneor more union labels have been applied.

“Visible spectrum” means the spectrum of frequencies that are less than790 THz and greater than 430 THz.

“White light image” means an image formed by white light in the visiblespectrum.

To say that a calculation is performed “with a classifier” means thatthe classifier performs the calculation.

Except to the extent that the context clearly requires otherwise, ifsteps in a method are described herein, then the method includesvariations in which: (1) steps in the method occur in any order orsequence, including any order or sequence different than that describedherein; (2) any step or steps in the method occurs more than once; (3)any two steps occur the same number of times or a different number oftimes during the method; (4) any combination of steps in the method isdone in parallel or serially; (5) any step in the method is performediteratively; (6) a given step in the method is applied to the same thingeach time that the given step occurs or is applied to different thingseach time that the given step occurs; (7) one or more steps occursimultaneously, or (8) the method includes other steps, in addition tothe steps described herein.

Headings are included herein merely to facilitate a reader's navigationof this document. A heading for a section does not affect the meaning orscope of that section.

This Definitions section shall, in all cases, control over and overrideany other definition of the Defined Terms. The Applicant or Applicantsare acting as his, her, its or their own lexicographer with respect tothe Defined Terms. For example, the definitions of Defined Terms setforth in this Definitions section override common usage or any externaldictionary. If a given term is explicitly or implicitly defined in thisdocument, then that definition shall be controlling, and shall overrideany definition of the given term arising from any source (e.g., adictionary or common usage) that is external to this document. If thisdocument provides clarification regarding the meaning of a particularterm, then that clarification shall, to the extent applicable, overrideany definition of the given term arising from any source (e.g., adictionary or common usage) that is external to this document. Unlessthe context clearly indicates otherwise, any definition or clarificationherein of a term or phrase applies to any grammatical variation of theterm or phrase, taking into account the difference in grammatical form.For example, the grammatical variations include noun, verb, participle,adjective, and possessive forms, and different declensions, anddifferent tenses.

Variations

This invention may be implemented in many different ways. Here are somenon-limiting examples:

In some implementations, this invention is a method comprising (a)creating a set of union-labeled modality A images in such a way thateach union-labeled image in the set is created by (i) capturing amodality A image, (ii) capturing a modality B image, (iii) acceptinginput, which input comprises expert labels for the modality A image,(iv) applying the expert labels to the modality A image on apixel-by-pixel basis, (v) performing a computer vision algorithm thatapplies computer vison labels to the modality B image on apixel-by-pixel basis, (vi) performing registration of the modality Aimage and modality B image, and (vii) after the registration, computingunion labels for the modality A image in such a way that, for eachspecific pixel of the modality A image, a union label for the specificpixel is a union of (A) the expert label for the specific pixel and (B)the computer vision label for a corresponding pixel of the modality Bimage; (b) training a classifier on at least the set of union-labeledmodality A images; and (c) after the training, calculating, with theclassifier, union labels for a previously unseen modality A image, on apixel-by-pixel basis. In some cases, each modality A image is a whitelight image and each modality B image is a fluorescent image In somecases, each modality A image and each modality B image is an image ofall or a portion of one or more teeth. In some cases, each expert label,computer vison label and union label indicates either presence of, orabsence of, dental plaque. In some cases, each expert label, computervison label and union label indicates either presence of, or absence of,gingivitis. In some cases, each modality A image and each modality Bimage is a tomographic image. In some cases, each union-labeled image inthe set captures a different physical object or different region of aphysical object than that which is captured in each other union-labeledimage in the set. Each of the cases described above in this paragraph isan example of the method described in the first sentence of thisparagraph, and is also an example of an embodiment of this inventionthat may be combined with other embodiments of this invention.

In some implementations, this invention is a method comprising: (a)creating a set of union-labeled modality A images in such a way thateach union-labeled image in the set is created by (i) capturing amodality A image, (ii) capturing a modality B image, (iii) applyinginitial labels to the modality A image on a pixel-by-pixel basis, (iv)applying initial labels to the modality B image on a pixel-by-pixelbasis, (v) performing registration of the modality A image and modalityB image, and (vi) after the registration, computing union labels for themodality A image in such a way that, for each specific pixel of themodality A image, a union label for the specific pixel is a union of (A)the initial label for the specific pixel and (B) the initial label for acorresponding pixel of the modality B image; (b) training a classifieron at least the set of union-labeled modality A images; and (c) afterthe training, calculating, with the classifier, union labels for apreviously unseen modality A image, on a region-by-region basis. In somecases, each union-labeled image in the set captures a different physicalobject or different region of a physical object than that which iscaptured in each other union-labeled image in the set. In some cases,for each particular union-labeled modality A image in the set: (a) theparticular union-labeled modality A image is an image of tissue beforethe tissue has been stained, and (b) a modality B image, whichcorresponds to the union-labeled modality A image, is an image of thetissue after the tissue has been stained. In some cases, for eachparticular union-labeled modality A image in the set: (a) the particularunion-labeled modality A image is an image of tissue before the tissuehas been stained by hematoxylin and eosin (H&E) stain, and (b) amodality B image, which corresponds to the union-labeled modality Aimage, is an image of the tissue after the tissue has been stained byH&E stain. In some cases, each modality A image and each modality Bimage is a tomographic image. In some cases: (a) each modality A imageis a positron emission tomography image; and (b) each modality B imageis a magnetic resonance imaging image. In some cases: (a) each modalityA image is a positron emission tomography image; and (b) each modality Bimage is an x-ray computed tomography image. Each of the cases describedabove in this paragraph is an example of the method described in thefirst sentence of this paragraph, and is also an example of anembodiment of this invention that may be combined with other embodimentsof this invention.

In some implementations, this invention is a system comprising: (a) amodality A imaging device; (b) a modality B imaging device; and (c) oneor more computers; wherein (i) the one or more computers are programmedto acquire a set of union-labeled modality A images, in such a way thatfor each specific union-labeled image in the set, the one or morecomputers are programmed (A) to output an instruction for the modality Aimaging device to capture a modality A image, (B) to output aninstruction for the modality B imaging device to capture a modality Bimage, (C) to accept input, which input comprises expert labels for themodality A image, (D) to apply the expert labels to the modality A imageon a pixel-by-pixel basis, (E) to perform a computer vision algorithmthat applies the computer vison labels to the modality B image on apixel-by-pixel basis, (F) to perform registration of the modality Aimage and modality B image, and (G) to compute, after the registration,union labels for the modality A image in such a way that, for eachspecific pixel of the modality A image, a union label for the specificpixel is a union of (I) the expert label for the specific pixel and (II)the computer vision label for a corresponding pixel of the modality Bimage, and (ii) the one or more computers are programmed (A) to train aclassifier on the set of union-labeled modality A images; and (B) afterthe training, to calculate, with the classifier, union labels for apreviously unseen modality A image, on a pixel-by-pixel basis. In somecases: (a) each modality A image is a white light image and eachmodality B image is a fluorescent image; and (b) each expert label,computer vison label and union label indicates either presence of, orabsence of, dental plaque. In some cases, each union-labeled image inthe set captures a different physical object or different region of aphysical object than that which is captured in each other union-labeledimage in the set. In some cases, for each particular union-labeledmodality A image in the set: (a) the particular union-labeled modality Aimage is an image of tissue before the tissue has been stained, and (b)a modality B image, which corresponds to the union-labeled modality Aimage, is an image of the tissue after the tissue has been stained. Insome cases, the modality A imaging device and the modality B imagingdevice are each a tomographic imaging device. In some cases: (a) themodality A imaging device is a positron emission tomography imagingdevice; and (b) the modality B imaging device is an x-ray computedtomography imaging device. In some cases: (a) each modality A image is awhite light image and each modality B image is a fluorescent image; and(b) each expert label, computer vison label and union label indicateseither presence of, or absence of, gingivitis. Each of the casesdescribed above in this paragraph is an example of the system describedin the first sentence of this paragraph, and is also an example of anembodiment of this invention that may be combined with other embodimentsof this invention.

In some implementations, this invention is an article of manufacturecomprising a non-transitory, machine-accessible medium havinginstructions encoded thereon: (i) for enabling one or more computers toperform the operations of (i) acquiring a set of union-labeled modalityA images, in such a way that for each specific union-labeled image inthe set, the one or more computers (A) output an instruction for amodality A imaging device to capture a modality A image, (B) output aninstruction for a modality B imaging device to capture a modality Bimage, (C) accept input, which input comprises expert labels for themodality A image, (D) apply the expert labels to the modality A image ona pixel-by-pixel basis, (E) perform a computer vision algorithm thatapplies the computer vison labels to the modality B image on apixel-by-pixel basis, (F) perform registration of the modality A imageand modality B image, and (G) compute, after the registration, unionlabels for the modality A image in such a way that, for each specificpixel of the modality A image, a union label for the specific pixel is aunion of (I) the expert label for the specific pixel and (II) thecomputer vision label for a corresponding pixel of the modality B image,and (ii) for enabling the one or more computers to perform theoperations of (A) training a classifier on the set of union-labeledmodality A images, and (B) after the training, to calculate, with theclassifier, union labels for a previously unseen modality A image, on apixel-by-pixel basis. In some cases: (a) each modality A image is awhite light image and each modality B image is a fluorescent image; and(b) each expert label, computer vison label and union label indicateseither presence of, or absence of, dental plaque. In some cases, eachunion-labeled image in the set captures a different physical object ordifferent region of a physical object than that which is captured ineach other union-labeled image in the set. In some cases, for eachparticular union-labeled modality A image in the set: (a) the particularunion-labeled modality A image is an image of tissue before the tissuehas been stained, and (b) a modality B image, which corresponds to theunion-labeled modality A image, is an image of the tissue after thetissue has been stained. In some cases, the modality A imaging deviceand the modality B imaging device are each a tomographic imaging device.In some cases: (a) the modality A imaging device is a positron emissiontomography imaging device; and (b) the modality B imaging device is anx-ray computed tomography imaging device. In some cases: (a) eachmodality A image is a white light image and each modality B image is afluorescent image; and (b) each expert label, computer vison label andunion label indicates either presence of, or absence of, gingivitis.Each of the cases described above in this paragraph is an example of thearticle of manufacture described in the first sentence of thisparagraph, and is also an example of an embodiment of this inventionthat may be combined with other embodiments of this invention.

In some implementations, this invention is a method comprisingparticipating in a download of software, either as a computer providingthe software or as a computer receiving the software, wherein thesoftware comprises instructions: (i) for enabling one or more computersto perform the operations of (i) acquiring a set of union-labeledmodality A images, in such a way that for each specific union-labeledimage in the set, the one or more computers (A) output an instructionfor a modality A imaging device to capture a modality A image, (B)output an instruction for a modality B imaging device to capture amodality B image, (C) accept input, which input comprises expert labelsfor the modality A image, (D) apply the expert labels to the modality Aimage on a pixel-by-pixel basis, (E) perform a computer vision algorithmthat applies the computer vison labels to the modality B image on apixel-by-pixel basis, (F) perform registration of the modality A imageand modality B image, and (G) compute, after the registration, unionlabels for the modality A image in such a way that, for each specificpixel of the modality A image, a union label for the specific pixel is aunion of (I) the expert label for the specific pixel and (II) thecomputer vision label for a corresponding pixel of the modality B image,and (ii) for enabling the one or more computers to perform theoperations of (A) training a classifier on the set of union-labeledmodality A images, and (B) after the training, to calculate, with theclassifier, union labels for a previously unseen modality A image, on apixel-by-pixel basis. In some cases: (a) each modality A image is awhite light image and each modality B image is a fluorescent image; and(b) each expert label, computer vison label and union label indicateseither presence of, or absence of, dental plaque. In some cases, eachunion-labeled image in the set captures a different physical object ordifferent region of a physical object than that which is captured ineach other union-labeled image in the set. In some cases, for eachparticular union-labeled modality A image in the set: (a) the particularunion-labeled modality A image is an image of tissue before the tissuehas been stained, and (b) a modality B image, which corresponds to theunion-labeled modality A image, is an image of the tissue after thetissue has been stained. In some cases, the modality A imaging deviceand the modality B imaging device are each a tomographic imaging device.In some cases: (a) the modality A imaging device is a positron emissiontomography imaging device; and (b) the modality B imaging device is anx-ray computed tomography imaging device. In some cases: (a) eachmodality A image is a white light image and each modality B image is afluorescent image; and (b) each expert label, computer vison label andunion label indicates either presence of, or absence of, gingivitis.Each of the cases described above in this paragraph is an example of themethod described in the first sentence of this paragraph, and is also anexample of an embodiment of this invention that may be combined withother embodiments of this invention.

Each description herein (or in the Provisional) of any method, apparatusor system of this invention describes a non-limiting example of thisinvention. This invention is not limited to those examples, and may beimplemented in other ways.

Each description herein (or in the Provisional) of any prototype of thisinvention describes a non-limiting example of this invention. Thisinvention is not limited to those examples, and may be implemented inother ways.

Each description herein (or in the Provisional) of any implementation,embodiment or case of this invention (or any use scenario for thisinvention) describes a non-limiting example of this invention. Thisinvention is not limited to those examples, and may be implemented inother ways.

Each Figure herein (or in the Provisional) that illustrates any featureof this invention shows a non-limiting example of this invention. Thisinvention is not limited to those examples, and may be implemented inother ways.

The above description (including without limitation any attacheddrawings and figures) describes illustrative implementations of theinvention. However, the invention may be implemented in other ways. Themethods and apparatus which are described herein are merely illustrativeapplications of the principles of the invention. Other arrangements,methods, modifications, and substitutions by one of ordinary skill inthe art are also within the scope of the present invention. Numerousmodifications may be made by those skilled in the art without departingfrom the scope of the invention. Also, this invention includes withoutlimitation each combination and permutation of one or more of thefeatures (including hardware, hardware components, methods, processes,steps, software, algorithms, features, or technology) that are describedherein.

What is claimed:
 1. A method comprising: (a) creating a set ofunion-labeled modality A images in such a way that each union-labeledimage in the set is created by (i) capturing a modality A image, (ii)capturing a modality B image, (iii) applying initial labels to themodality A image on a region-by-region basis, (iv) applying initiallabels to the modality B image on a region-by-region basis, (v)performing registration of the modality A image and modality B image,and (vi) after the registration, computing union labels for the modalityA image in such a way that, for each specific region in a group ofregions of the modality A image, a union label for the specific regionis a union of (A) the initial label for the specific region and (B) theinitial label for a corresponding region of the modality B image; (b)training a classifier on at least the set of union-labeled modality Aimages; and (c) after the training, calculating, with the classifier,labels for a previously unseen modality A image, on a region-by-regionbasis.
 2. The method of claim 1, wherein each union-labeled image in theset captures a different physical object or different portion of aphysical object than that which is captured in each other union-labeledimage in the set.
 3. The method of claim 1, wherein, for each particularunion-labeled modality A image in the set: (a) the particularunion-labeled modality A image is an image of tissue before the tissuehas been stained, and (b) a modality B image, which corresponds to theunion-labeled modality A image, is an image of the tissue after thetissue has been stained.
 4. The method of claim 1, wherein, for eachparticular union-labeled modality A image in the set: (a) the particularunion-labeled modality A image is an image of tissue before the tissuehas been stained by hematoxylin and eosin (H&E) stain, and (b) amodality B image, which corresponds to the union-labeled modality Aimage, is an image of the tissue after the tissue has been stained byH&E stain.
 5. The method of claim 1, wherein each modality A image andeach modality B image is a tomographic image.
 6. The method of claim 1,wherein: (a) each modality A image is a positron emission tomographyimage; and (b) each modality B image is a magnetic resonance imagingimage.
 7. The method of claim 1, wherein: (a) each modality A image is apositron emission tomography image; and (b) each modality B image is anx-ray computed tomography image.
 8. The method of claim 1, wherein eachmodality A image is a white light image and each modality B image is afluorescent image.
 9. The method of claim 1, wherein each modality Aimage and each modality B image is an image of all or a portion of oneor more teeth.
 10. The method of claim 1, wherein each initial label andeach union label indicates either presence of, or absence of, dentalplaque.
 11. The method of claim 1, wherein each initial label and eachunion label indicates either presence of, or absence of, gingivitis. 12.The method of claim 1, wherein: (a) each initial label of a modality Aimage is inputted by a human being; and (b) each initial label of amodality B image is generated by a computer vision algorithm.
 13. Themethod of claim 1, wherein each region mentioned in claim 1 consists ofonly a single pixel.
 14. The method of claim 1, wherein each regionmentioned in claim 1 consists of multiple pixels.
 15. A systemcomprising: (a) a modality A imaging device; (b) a modality B imagingdevice; and (c) one or more computers; wherein (i) the one or morecomputers are programmed to acquire a set of union-labeled modality Aimages, in such a way that for each specific union-labeled image in theset, the one or more computers are programmed (A) to output aninstruction for the modality A imaging device to capture a modality Aimage, (B) to output an instruction for the modality B imaging device tocapture a modality B image, (C) to apply initial labels to the modalityA image on region-by-region basis, (D) to apply initial labels to themodality B image on a region-by-region basis, (E) to performregistration of the modality A image and modality B image, and (F) tocompute, after the registration, union labels for the modality A imagein such a way that, for each specific region in a group of regions ofthe modality A image, a union label for the specific region is a unionof (I) the initial label for the specific region and (II) the initiallabel for a corresponding region of the modality B image, and (ii) theone or more computers are also programmed (A) to train a classifier onthe set of union-labeled modality A images; and (B) after the training,to calculate, with the classifier, labels for a previously unseenmodality A image, on a region-by-region basis.
 16. The system of claim15, wherein: (a) each modality A image is a white light image and eachmodality B image is a fluorescent image; and (b) each initial label andunion label indicates either presence of, or absence of, dental plaque.17. The system of claim 15, wherein each union-labeled image in the setcaptures a different physical object or different region of a physicalobject than that which is captured in each other union-labeled image inthe set.
 18. The system of claim 15, wherein, for each particularunion-labeled modality A image in the set: (a) the particularunion-labeled modality A image is an image of tissue before the tissuehas been stained, and (b) a modality B image, which corresponds to theunion-labeled modality A image, is an image of the tissue after thetissue has been stained.
 19. The method of claim 15, wherein the one ormore computers are also programmed: (a) to accept input from one or morehuman beings, which input specifies the initial labels for the modalityA images; (b) to generate, by a computer vision algorithm, the initiallabels for the modality B images.
 20. The system of claim 15 wherein themodality A imaging device and the modality B imaging device are each atomographic imaging device.
 21. The system of claim 15, wherein: (a)each modality A image is a white light image and each modality B imageis a fluorescent image; and (b) each initial label and union labelindicates either presence of, or absence of, gingivitis.